-- Logs begin at Thu 2022-09-29 01:29:37 UTC, end at Thu 2022-09-29 02:28:19 UTC. -- Sep 29 01:56:57.295369 dv-cinder-740146-21-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 29 01:56:57.303861 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Sep 29 01:56:57 2022] *** Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: nodename: dv-cinder-740146-21-iscsi Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: machine: x86_64 Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: clock source: unix Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: pcre jit disabled Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: detected number of CPU cores: 4 Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: current working directory: / Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: detected binary path: /usr/bin/uwsgi-core Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: your processes number limit is 47680 Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: your memory page size is 4096 bytes Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: detected max file descriptor number: 2048 Sep 29 01:56:57.306483 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: lock engine: pthread robust mutexes Sep 29 01:56:57.307366 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: thunder lock: enabled Sep 29 01:56:57.307366 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 29 01:56:57.307366 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 29 01:56:57.345466 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: Python main interpreter initialized at 0x55a43abf29c0 Sep 29 01:56:57.345466 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: python threads support enabled Sep 29 01:56:57.345466 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: your server socket listen backlog is limited to 100 connections Sep 29 01:56:57.345466 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: your mercy for graceful operations on workers is 90 seconds Sep 29 01:56:57.345794 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: mapped 403077 bytes (393 KB) for 2 cores Sep 29 01:56:57.345893 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: *** Operational MODE: preforking *** Sep 29 01:56:57.346157 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: *** uWSGI is running in multiple interpreter mode *** Sep 29 01:56:57.346157 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: spawned uWSGI master process (pid: 81535) Sep 29 01:56:57.346341 dv-cinder-740146-21-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 29 01:56:57.347387 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: spawned uWSGI worker 1 (pid: 81543, cores: 1) Sep 29 01:56:57.347387 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: spawned uWSGI worker 2 (pid: 81544, cores: 1) Sep 29 01:56:57.347387 dv-cinder-740146-21-iscsi devstack@keystone.service[81535]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 29 01:57:01.624378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:57:01.624378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: from cryptography.utils import int_from_bytes Sep 29 01:57:01.625088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:57:01.625088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: from cryptography.utils import int_from_bytes Sep 29 01:57:01.626223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:57:01.626223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: from cryptography.utils import int_from_bytes Sep 29 01:57:01.626669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:57:01.626669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: from cryptography.utils import int_from_bytes Sep 29 01:57:01.675403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 29 01:57:01.675595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 29 01:57:01.675789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] command line args: [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 29 01:57:01.675876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 29 01:57:01.676021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 29 01:57:01.676105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 29 01:57:01.676214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] command line args: [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 29 01:57:01.676292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ================================================================================ {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 29 01:57:01.676468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] admin_token = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.676545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 29 01:57:01.676646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ================================================================================ {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 29 01:57:01.676731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] config_dir = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.676916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] admin_token = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] config_source = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] config_dir = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] debug = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] config_source = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] debug = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.677975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_config_append = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.678943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_config_append = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_dir = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_file = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_dir = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_file = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.679919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.680995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.681944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.682890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683316 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] publish_errors = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] publish_errors = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.683965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.684943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] standard_threads = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] standard_threads = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] use_journal = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] use_json = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.685970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] use_journal = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] use_stderr = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] use_json = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] use_syslog = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] use_stderr = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] use_syslog = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:57:01.686668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.686835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.686918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.687334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.687413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.687593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.687671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.687831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.687910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.688859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.689843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.690962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.691923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.692945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.693956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.694124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.694306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.694474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.695210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.696991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.698748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.698974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.699189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.699930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.700972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.701985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.702954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.703907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.704984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.705932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.706946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707152 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.707927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.708943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.709971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.710881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.711014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.711135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.711296 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.711426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.712941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.external = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.external = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.713930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.password = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] auth.token = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.password = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] auth.token = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.714963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.715907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.716909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.717864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.718825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.719861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.720978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.721954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.722920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.723938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.724937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.725853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.726854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.727971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.728932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.729972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.730909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.731993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.732968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.733873 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.734982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.735924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.736918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.737935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.738883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.739853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.740974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.741928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742171 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.742977 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.743926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.744894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.745875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] role.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] role.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.746964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] role.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] role.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.747937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.748940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.749992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.750148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.750230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.750395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.750539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.750644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.750776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.751995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.752975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.753994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.754955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.755897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756783 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.756863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.757848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.758002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.758582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.758774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.758954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.759033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.759264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.759355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.759496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.759706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.759787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.connection = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.760973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.761197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.761305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.761432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.761674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.761755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.connection = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.761909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.762920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.763962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.764043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.764240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.764320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.764410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81543) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 29 01:57:01.764661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.765028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.765343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.765660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.765970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.766289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.766596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.766912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.767254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.767565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.767869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:57:01.768106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81544) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 29 01:57:01.794157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.794386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.794613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.796552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.796786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.797037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.802923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.805266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.831420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.831520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.831645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.831729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.889024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.891066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.937780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.937905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.938011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:01.938096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.093957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.094032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.094107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.094211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.094305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.094431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.117079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.117312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.117398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.118916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.119000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.119130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.129972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.130340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.142554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.142734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.142960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81544) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.143914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.144045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.144264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81543) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:57:02.160786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.161022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:57:02.161364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.161489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.161572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:57:02.161700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:57:02.161822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.162043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.162126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.162242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:57:02.162637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.162723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.162798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.162907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.163023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.163255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.174179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.174463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.174737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.175026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.175503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.175784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.176054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.176338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.183276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.185252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.186212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.186501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.186788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.187327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.187614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.187880 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.188154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.188383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.188465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.188666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.188749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.188968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.189063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.189242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.189323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.189521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.189781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.190050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.190321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.190590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.190862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.191128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.212342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.213852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.214100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.214182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.215557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.215840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.218797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.219085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.219371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.220475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.220757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.221062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.225718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.226007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.227907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.228187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.232219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.232516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.232805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.233116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.233397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.233675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.233820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.233947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.234116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.234203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.234406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.234695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.235033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.235244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.235510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.235776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.249746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.250025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.251211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.251488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.253909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.255329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.257156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.257459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.259090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.259385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.262237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.263583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.265386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.266763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.268603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.268932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.269222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.269507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.269796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.269968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.270086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.270266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.270547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.270824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.271107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.271391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.285910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.286205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.286475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.286738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.287024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.287313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.287582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.287843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.294147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.295148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.295698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.296680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.297249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.297529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.298230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.298505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.300238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.300527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.300814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.301123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.301206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.301407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.301505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.301689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.301970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.302363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.312938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.313228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.313514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.313704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.313786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.313988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.314172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.314255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.314469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.314552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.314754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.314893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.315180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.315459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.330597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.331217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.333532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.334130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.336530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.337057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.338054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.338484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.339524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.339811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.339947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.340214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.340296 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.340614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.346601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.346948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.348078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.348408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.351006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.351293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.351408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.351687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.351782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.352504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.352651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.352780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.360294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.360513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.360596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.360802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.360905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.361114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.367786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81543) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.367981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81544) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:57:02.368079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.368264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.368348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.368543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.368626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.368825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.368948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.369940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.370889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.371043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.371125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.371304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.371386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81543) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.371569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81544) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:57:02.405598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55a43abf29c0 pid: 81544 (default app) Sep 29 01:57:02.405695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55a43abf29c0 pid: 81543 (default app) Sep 29 01:57:02.408767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb7b12-701e-4b42-982b-83d5f8ec83e2 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:02.409076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb7b12-701e-4b42-982b-83d5f8ec83e2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:02.409350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb7b12-701e-4b42-982b-83d5f8ec83e2 None None] PATH_INFO: `/v3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:02.413459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Thu Sep 29 01:57:02 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:09.151748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:09.152257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:09.152257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:09.156486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:09 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:09.161777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:09.162330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:09.162330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:09.236194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: INFO dbcounter [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] Registered counter for database keystone Sep 29 01:57:09.254981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG oslo_db.sqlalchemy.engines [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81544) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 29 01:57:09.261704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writer thread running {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 29 01:57:09.441166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG passlib.handlers.bcrypt [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81544) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 29 01:57:09.441479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG passlib.handlers.bcrypt [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81544) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 29 01:57:09.498694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:09.500269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:09.625189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:09 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 29 01:57:10.105214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:10.176091 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: INFO dbcounter [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] Registered counter for database keystone Sep 29 01:57:10.197227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG oslo_db.sqlalchemy.engines [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81543) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 29 01:57:10.203383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writer thread running {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 29 01:57:10.320012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5bfc6-d16f-43cb-999b-f5fcaf7151f4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:10.324261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:10.324893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:10.325186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:10.325456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:10.333488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:10.384987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING keystone.common.rbac_enforcer.enforcer [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 29 01:57:10.385276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b07f42-adba-4878-a6ca-4536c9230e87 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:10.394292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Sep 29 01:57:10 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 01:57:10.400739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:10.417668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa59ca7-64c8-408f-9b14-a836f03d11b4 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:10.422023 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:10.423110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:10.423389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:10.423664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] PATH_INFO: `/v3/projects/304c048daacb48678f2d64fc933b378e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:10.425237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] RBAC: Authorizing `identity:get_project(project_id=304c048daacb48678f2d64fc933b378e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:10.478323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING keystone.common.rbac_enforcer.enforcer [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 29 01:57:10.478668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d298792-4f5f-4667-859b-5d5337fb6ead admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:10.483164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Sep 29 01:57:10 2022] GET /identity/v3/projects/304c048daacb48678f2d64fc933b378e => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:13.011974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36bf16f-7e01-46a3-970d-91bcbac96e48 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:13.012506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36bf16f-7e01-46a3-970d-91bcbac96e48 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:13.012506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36bf16f-7e01-46a3-970d-91bcbac96e48 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:13.016623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:13 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:13.022129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:13.022504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:13.022686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:13.042734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:13.044227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:13.052026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:13 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 29 01:57:13.329512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36bf16f-7e01-46a3-970d-91bcbac96e48 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:13.341973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36bf16f-7e01-46a3-970d-91bcbac96e48 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:13.346145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22d8271-3642-4a17-9f66-cab8941b1bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:13.346740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22d8271-3642-4a17-9f66-cab8941b1bd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:13.347023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22d8271-3642-4a17-9f66-cab8941b1bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:13.347299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22d8271-3642-4a17-9f66-cab8941b1bd5 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:13.354966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22d8271-3642-4a17-9f66-cab8941b1bd5 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:13.356745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22d8271-3642-4a17-9f66-cab8941b1bd5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:13.366793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Sep 29 01:57:13 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 29 01:57:13.373715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:13.382867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7617b6ba-783f-49f6-9e7a-8a5adfa17241 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:13.387811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ed40b6-341a-4a20-84c4-d533f8848b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:13.388440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed40b6-341a-4a20-84c4-d533f8848b27 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:13.388726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed40b6-341a-4a20-84c4-d533f8848b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:13.389046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed40b6-341a-4a20-84c4-d533f8848b27 admin admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:13.390369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ed40b6-341a-4a20-84c4-d533f8848b27 admin admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:13.392196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ed40b6-341a-4a20-84c4-d533f8848b27 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:13.393765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Thu Sep 29 01:57:13 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:15.586086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab056b80-eff9-400a-8b5f-9129c937fe1b None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:15.586961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab056b80-eff9-400a-8b5f-9129c937fe1b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:15.587067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab056b80-eff9-400a-8b5f-9129c937fe1b None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:15.588309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:15.593959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:15.594386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:15.594644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:15.613115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:15.614309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:15.614414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de36aea-61a3-42c6-b514-481117b3ba37 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:15.614673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de36aea-61a3-42c6-b514-481117b3ba37 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:15.614948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de36aea-61a3-42c6-b514-481117b3ba37 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:15.616248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:15.621328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:15.621614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:15.621890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:15.663646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG passlib.handlers.bcrypt [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81543) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 29 01:57:15.664044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG passlib.handlers.bcrypt [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81543) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 29 01:57:15.694558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:15.714423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:15.716027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:15.815119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:15.960896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:15.984433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.075487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3967dc09-0db8-40d1-89e4-c028311ec791 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.084128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.108125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bef256-f59f-4daf-9313-796153194518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.108756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bef256-f59f-4daf-9313-796153194518 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.109064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bef256-f59f-4daf-9313-796153194518 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.109335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bef256-f59f-4daf-9313-796153194518 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.120626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bef256-f59f-4daf-9313-796153194518 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-79bef256-f59f-4daf-9313-796153194518 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:16.129356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:16.130268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:16.131097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:16.136186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bef256-f59f-4daf-9313-796153194518 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.185572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652340ba-1bbf-48a0-aea8-e92beaf05a67 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.188037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.217974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.222693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.223551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.223834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.224106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.237319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bef256-f59f-4daf-9313-796153194518 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.269728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.270335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.270624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.270931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.271599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:16.272466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.273913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.291849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.293481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.294125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:16.299727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.393901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:57:16.397267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844eb2ba-b313-4134-aa4d-f26da5c55a6c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.399574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.402307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.432166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.432805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.433123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.433401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.434555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.436033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.437637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:16.443334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.457690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.498319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706135e-5d0f-45e0-8d23-cc42b9967b3b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.531533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.532393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.532679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.532980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.540439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcca261-7925-477d-ae1a-4ba377b3d0c6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.551630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.553197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:57:16.571719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:16.572683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:57:16.573525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:57:16.573525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:57:16.573525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:16.573525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 9/18] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:16.578703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.579585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.580217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.580499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.580775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.581940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.583398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.585009 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 10/19] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:16.590723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:16.634515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.669903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:16.685843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa992a9-7c98-44e1-9b93-1d050c16210c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.691422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfadfff-912e-4e66-97d7-c78ccb2aec21 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:16.721613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.722236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.722522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.722867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.723517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:16.723826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.725246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0df0e-a543-4d43-8c91-15dcd7299606 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.727910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:16.728520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:16.728800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:16.729103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:16.729849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.domain.id=default, user.id=90cfae9be7734268a0ec1a4bc2e21b77) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:16.730125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:16.731552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94e4249-4843-4187-b1f7-04315bb310c3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:16.738041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 11/20] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/role_assignments?user.id=90cfae9be7734268a0ec1a4bc2e21b77&scope.domain.id=default&role.id=619e79e7bb94488780bd4da040d61c34 => generated 233 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:16.743493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 10/21] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:57:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:19.271172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96328ac3-6164-460d-884e-cb398113dd4f None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.272172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96328ac3-6164-460d-884e-cb398113dd4f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.272674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96328ac3-6164-460d-884e-cb398113dd4f None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.275045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 12/22] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:19.281607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6c2f2d-846b-4a06-919a-3e8583762680 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.282059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6c2f2d-846b-4a06-919a-3e8583762680 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.282215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6c2f2d-846b-4a06-919a-3e8583762680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.306558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cc6c2f2d-846b-4a06-919a-3e8583762680 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:19.307901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cc6c2f2d-846b-4a06-919a-3e8583762680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:19.350964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4710e255-990e-443c-8f18-c70426d90a44 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.351638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4710e255-990e-443c-8f18-c70426d90a44 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.352102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4710e255-990e-443c-8f18-c70426d90a44 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.353734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 13/23] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:19.366182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb6c11b-95ff-45e6-b424-90de904930ef None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.366783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb6c11b-95ff-45e6-b424-90de904930ef None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.367263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb6c11b-95ff-45e6-b424-90de904930ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.386876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 11/24] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:19.408054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-beb6c11b-95ff-45e6-b424-90de904930ef None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:19.409329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-beb6c11b-95ff-45e6-b424-90de904930ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:19.498220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 14/25] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:19.537694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99618a77-73ce-4534-b81a-707a9babf28f None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.538294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99618a77-73ce-4534-b81a-707a9babf28f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.538762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99618a77-73ce-4534-b81a-707a9babf28f None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.540888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 12/26] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:19.546357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.546870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.547348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.566617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:19.568103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:19.660091 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99618a77-73ce-4534-b81a-707a9babf28f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:19.672804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 15/27] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:19.720782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:19.782797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99618a77-73ce-4534-b81a-707a9babf28f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:19.816824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:19.817401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:19.817684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:19.817953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:19.832296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:19.833832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:57:19.850642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:19.851494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:57:19.852810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:57:19.852810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:57:19.852810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:19.854087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 13/28] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:57:19.859947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:19.865211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:19.934119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:20.014377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f837a19a-269a-44ed-9f7a-fff42a3264ce None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:20.082238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:20.082782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.083068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.083338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.083985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:20.084286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:20.085702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc14d22-c11e-43c5-bf0a-eb6037931b9b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:20.108024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:20.113681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d26d03b-63d6-4ddf-ad6e-01d97d15c6ad None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:20.123423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 16/29] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:57:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:20.124755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca492d9-870d-46e9-99fe-5b0e6f7de7e9 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.125067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca492d9-870d-46e9-99fe-5b0e6f7de7e9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.125335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca492d9-870d-46e9-99fe-5b0e6f7de7e9 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.126687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 17/30] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:20.130542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca492d9-870d-46e9-99fe-5b0e6f7de7e9 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:20.155646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5fd708-7d36-4edc-ae0a-65f9a34c2832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:20.156430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5fd708-7d36-4edc-ae0a-65f9a34c2832 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.156781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5fd708-7d36-4edc-ae0a-65f9a34c2832 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.157350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5fd708-7d36-4edc-ae0a-65f9a34c2832 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.171274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5fd708-7d36-4edc-ae0a-65f9a34c2832 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7c5fd708-7d36-4edc-ae0a-65f9a34c2832 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:20.179152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:20.179981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:20.181295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 14/31] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:57:20.182161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.182653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.183129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.203356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:20.204955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:20.217603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:20.263744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca492d9-870d-46e9-99fe-5b0e6f7de7e9 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:20.288490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 15/32] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:20.289551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:20.330933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:20.331525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.331804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.332125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.332838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:20.334516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:20.452752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:20.475566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8634c197-e3e7-4fc0-87df-8d2ad456fcac None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:20.542486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:20.543460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.543960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.544437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.545381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:20.546951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:20.567211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 18/33] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:57:20 2022] POST /identity/v3/projects => generated 301 bytes in 437 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:57:20.568515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:20.708417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 16/34] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 29 01:57:20 2022] POST /identity/v3/roles => generated 219 bytes in 419 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:20.709468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:20.777893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adf745b-b983-4af5-b53d-b169bc449e0f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:20.810674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:20.851862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:20.852473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:20.852756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:20.853057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:20.853735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:20.854037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:20.855642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:20.873694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 19/35] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:20.879753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:20.978397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc6612c-0bcc-47ad-8012-11e7cddcafd3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:21.028515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fc6008-3ba0-414d-86a0-9983238db8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:21.029590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fc6008-3ba0-414d-86a0-9983238db8db None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.030089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fc6008-3ba0-414d-86a0-9983238db8db None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.030581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fc6008-3ba0-414d-86a0-9983238db8db None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.031338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fc6008-3ba0-414d-86a0-9983238db8db None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:21.033094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fc6008-3ba0-414d-86a0-9983238db8db None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:21.044556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b61be90-3bc3-4f50-96a0-3c9a826e158c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:21.054112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 17/36] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 29 01:57:20 2022] POST /identity/v3/roles => generated 225 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:21.055667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f9c225-ca96-4aa4-b65f-130e9e71ce71 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.056077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f9c225-ca96-4aa4-b65f-130e9e71ce71 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.056362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f9c225-ca96-4aa4-b65f-130e9e71ce71 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.057820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 18/37] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:21.065109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.065679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.066099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.088782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:21.089060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:21.089936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.090387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.090494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:21.090931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.092624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:21.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:21.095312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:21.098036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 20/38] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:20 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:21.106087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:21.177354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:21.185505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 19/39] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:21.248351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb98646b-0a68-457c-8829-02d123e84113 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:21.287077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d8eac2-90db-45cc-a766-01a0e39931b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:21.288063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d8eac2-90db-45cc-a766-01a0e39931b4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.288544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d8eac2-90db-45cc-a766-01a0e39931b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.289066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d8eac2-90db-45cc-a766-01a0e39931b4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.291053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d8eac2-90db-45cc-a766-01a0e39931b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:21.292999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d8eac2-90db-45cc-a766-01a0e39931b4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:21.295076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 21/40] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:21.300504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:21.333976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1835ae9-5b98-44cf-b756-a246718bcf77 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.334479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1835ae9-5b98-44cf-b756-a246718bcf77 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.335008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1835ae9-5b98-44cf-b756-a246718bcf77 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.337499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 22/41] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:21.343304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.343594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.343873 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.423432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fd145-1b2d-402c-b0bb-a6c3efcedc8d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:21.460239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:21.461315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.462335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.462817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] PATH_INFO: `/v3/domains/default/users/90cfae9be7734268a0ec1a4bc2e21b77/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.466341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=90cfae9be7734268a0ec1a4bc2e21b77, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:21.468007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:21.470378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:21.471897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:21.479897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:21.485422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 20/42] 10.222.0.27 () {58 vars in 1361 bytes} [Thu Sep 29 01:57:21 2022] PUT /identity/v3/domains/default/users/90cfae9be7734268a0ec1a4bc2e21b77/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:57:21.486816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:21.554810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 23/43] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:21.610019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02a2f5c-4d23-4d11-bb14-dcbc5b2191a1 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:21.646605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:21.647241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.647526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.647803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.648487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:21.650479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:21.669113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 21/44] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 29 01:57:21 2022] POST /identity/v3/roles => generated 223 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:21.830295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:21.878505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:21.933526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df1e2d-3958-4508-a88c-73e83477fedd None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:21.967033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc291bdf-06c4-4cfa-972e-e1cffd3e122a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:21.967666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc291bdf-06c4-4cfa-972e-e1cffd3e122a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:21.967950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc291bdf-06c4-4cfa-972e-e1cffd3e122a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:21.968228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc291bdf-06c4-4cfa-972e-e1cffd3e122a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:21.969357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc291bdf-06c4-4cfa-972e-e1cffd3e122a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:21.970803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc291bdf-06c4-4cfa-972e-e1cffd3e122a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:21.972361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 24/45] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:21.978192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:22.035237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:22.076248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14445c61-1c2d-4de0-be25-e3210432f035 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:22.108289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5eb54f-1c89-4d2b-820a-ae37ab1b271b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:22.108836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5eb54f-1c89-4d2b-820a-ae37ab1b271b None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:22.109147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5eb54f-1c89-4d2b-820a-ae37ab1b271b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:22.109422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5eb54f-1c89-4d2b-820a-ae37ab1b271b None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:22.110064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5eb54f-1c89-4d2b-820a-ae37ab1b271b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:22.111513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5eb54f-1c89-4d2b-820a-ae37ab1b271b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:22.139098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 22/46] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:57:21 2022] POST /identity/v3/projects => generated 312 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:23.639090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f646e-7e5d-4d6a-bb38-35d410899037 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:23.639636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f646e-7e5d-4d6a-bb38-35d410899037 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:23.639636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f646e-7e5d-4d6a-bb38-35d410899037 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:23.640992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 25/47] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:23.646214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:23.646598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:23.646778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:23.664775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:23.666507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:23.730054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 23/48] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:23.999545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f646e-7e5d-4d6a-bb38-35d410899037 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.096175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f646e-7e5d-4d6a-bb38-35d410899037 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.109612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:24.130650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590089e5-1101-44c4-bb27-95c5f291c147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.131286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590089e5-1101-44c4-bb27-95c5f291c147 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.131570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590089e5-1101-44c4-bb27-95c5f291c147 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.131848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590089e5-1101-44c4-bb27-95c5f291c147 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.142228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590089e5-1101-44c4-bb27-95c5f291c147 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-590089e5-1101-44c4-bb27-95c5f291c147 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:24.147088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:24.148050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:24.148794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 26/49] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:24.154266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.200062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:24.260160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de32d4f-16b7-4ba3-a82e-c4e9b380a218 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.293327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.293938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.294225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.294500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.295163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:24.295471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:24.296861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:24.303266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcac2bc-1f9f-4831-bc50-25c905fa7bca None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.303549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcac2bc-1f9f-4831-bc50-25c905fa7bca None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.303824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcac2bc-1f9f-4831-bc50-25c905fa7bca None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.305172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 27/50] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:24.309992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.310279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.310554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.312997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 24/51] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:24.318779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.328847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:24.330056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:24.397973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 28/52] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:24.518868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5dc8d2-146b-4725-b537-d4662f0d24ab None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.520253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:24.551439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.552065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.552345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.552619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.553763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:24.555200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:24.557320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 25/53] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:24 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:24.562745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.665302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d545d51-2703-4fb3-af9c-b41afebb38a7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.666776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:24.668980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.700321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.700996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.701284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.701565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.702681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:24.704140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:24.705959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 29/54] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:24.711377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.767265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d5ca17-8945-459f-8c13-9b6d3500ade0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.800261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.800817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.801129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.801415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.802536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:24.803980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:24.806170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 26/55] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:24.809571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fa23aa-49f2-4053-ab4d-d55264642439 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.811626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:24.842727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.843420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.843697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.843970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.844681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.domain.id=default, user.id=90cfae9be7734268a0ec1a4bc2e21b77) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:24.845020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:24.846465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a6241b-83ec-4c13-9645-3d981f9da898 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:24.850562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:24.852119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 30/56] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Sep 29 01:57:24 2022] GET /identity/v3/role_assignments?user.id=90cfae9be7734268a0ec1a4bc2e21b77&scope.domain.id=default&role.id=619e79e7bb94488780bd4da040d61c34 => generated 530 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:24.916360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475d338e-c02f-4911-943b-5ccf0dde4ea8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:24.953050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d495377a-0303-4033-bfea-a2c1879b9fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:24.953676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d495377a-0303-4033-bfea-a2c1879b9fc3 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:24.953964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d495377a-0303-4033-bfea-a2c1879b9fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:24.954243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d495377a-0303-4033-bfea-a2c1879b9fc3 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:24.954909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d495377a-0303-4033-bfea-a2c1879b9fc3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:24.956377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d495377a-0303-4033-bfea-a2c1879b9fc3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:24.984452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 27/57] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Sep 29 01:57:24 2022] POST /identity/v3/projects => generated 298 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:57:27.121080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003decf-61da-4511-9cf0-0e80a629c953 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:27.121572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003decf-61da-4511-9cf0-0e80a629c953 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:27.121653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003decf-61da-4511-9cf0-0e80a629c953 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:27.122951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 31/58] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:27.128116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:27.128544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:27.128676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:27.146868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:27.148065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:27.212593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 28/59] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:27.482163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003decf-61da-4511-9cf0-0e80a629c953 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:27.515471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:27.579394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003decf-61da-4511-9cf0-0e80a629c953 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:27.612252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522d5e0b-172f-4699-84a2-2809bbbb41bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:27.612888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522d5e0b-172f-4699-84a2-2809bbbb41bc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:27.613171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522d5e0b-172f-4699-84a2-2809bbbb41bc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:27.613443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522d5e0b-172f-4699-84a2-2809bbbb41bc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:27.614537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522d5e0b-172f-4699-84a2-2809bbbb41bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:27.615970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522d5e0b-172f-4699-84a2-2809bbbb41bc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:27.617541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 32/60] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:27.622975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:27.643324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:27.717507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58359977-df3c-46da-a4e5-2bd98272203b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:27.749337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45353193-3605-4c86-8d25-509be280206d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:27.749937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45353193-3605-4c86-8d25-509be280206d None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:27.750220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45353193-3605-4c86-8d25-509be280206d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:27.750494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45353193-3605-4c86-8d25-509be280206d None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:27.751145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45353193-3605-4c86-8d25-509be280206d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:27.752585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45353193-3605-4c86-8d25-509be280206d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:27.779665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 29/61] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:27 2022] POST /identity/v3/users => generated 275 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:30.893191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d009d97f-f6c9-485d-a463-98f5284d1f55 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:30.894163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d009d97f-f6c9-485d-a463-98f5284d1f55 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:30.894683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d009d97f-f6c9-485d-a463-98f5284d1f55 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:30.896253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 33/62] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:30.907299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f85a8e-4caa-485c-a2f9-199cb9ab26ad None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:30.908268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f85a8e-4caa-485c-a2f9-199cb9ab26ad None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:30.908745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f85a8e-4caa-485c-a2f9-199cb9ab26ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:30.935172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-08f85a8e-4caa-485c-a2f9-199cb9ab26ad None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:30.936394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-08f85a8e-4caa-485c-a2f9-199cb9ab26ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:31.024920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 30/63] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:31.358614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c507b1d-4d41-4225-b6af-f27b77678d78 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.359283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c507b1d-4d41-4225-b6af-f27b77678d78 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.359756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c507b1d-4d41-4225-b6af-f27b77678d78 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.361360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 34/64] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:31.366761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e076e9-c105-423d-82ac-359de8381a2d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.367055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e076e9-c105-423d-82ac-359de8381a2d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.367345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e076e9-c105-423d-82ac-359de8381a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.380445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:31.390055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-28e076e9-c105-423d-82ac-359de8381a2d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:31.392220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-28e076e9-c105-423d-82ac-359de8381a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:31.431294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9cb94-056c-4956-a9da-e4a9ed7f64c1 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.431578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9cb94-056c-4956-a9da-e4a9ed7f64c1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.431857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b9cb94-056c-4956-a9da-e4a9ed7f64c1 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.434062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 35/65] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:31.434837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b9cb94-056c-4956-a9da-e4a9ed7f64c1 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:31.480618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 31/66] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:31.482445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4a0b9d-ed4b-4212-a11f-32bec318ac6c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.482923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4a0b9d-ed4b-4212-a11f-32bec318ac6c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.483363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4a0b9d-ed4b-4212-a11f-32bec318ac6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.514370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4b4a0b9d-ed4b-4212-a11f-32bec318ac6c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:31.517620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4b4a0b9d-ed4b-4212-a11f-32bec318ac6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:31.609810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b9cb94-056c-4956-a9da-e4a9ed7f64c1 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:31.613272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:31.650541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 32/67] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:31.652021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09919ab-1052-41d1-932f-3bf50cd479d1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.652444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09919ab-1052-41d1-932f-3bf50cd479d1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.652852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09919ab-1052-41d1-932f-3bf50cd479d1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.654291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 33/68] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:31.662362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02381b83-1fe5-43d5-9439-6f058467f1d2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.662834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02381b83-1fe5-43d5-9439-6f058467f1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.663243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02381b83-1fe5-43d5-9439-6f058467f1d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.670698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f9a97e-5b9b-4a35-bbfc-da972a0b12a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:31.673127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9a97e-5b9b-4a35-bbfc-da972a0b12a4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.673563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9a97e-5b9b-4a35-bbfc-da972a0b12a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.673988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9a97e-5b9b-4a35-bbfc-da972a0b12a4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.697696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f9a97e-5b9b-4a35-bbfc-da972a0b12a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:31.698832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-02381b83-1fe5-43d5-9439-6f058467f1d2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:31.704532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-02381b83-1fe5-43d5-9439-6f058467f1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-33f9a97e-5b9b-4a35-bbfc-da972a0b12a4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:31.714148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:31.715039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:31.716181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 36/69] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:31.722826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5ece32-1636-43b5-9262-fc5f18b1b894 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.722826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5ece32-1636-43b5-9262-fc5f18b1b894 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.722826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5ece32-1636-43b5-9262-fc5f18b1b894 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.723637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 37/70] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:31.726670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5ece32-1636-43b5-9262-fc5f18b1b894 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:31.782681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:31.792826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 34/71] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:31.794555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.794861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.795279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.820466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:31.822294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:31.837494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5ece32-1636-43b5-9262-fc5f18b1b894 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:31.872613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:31.873592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:31.874098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:31.874562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:31.875412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:31.875905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:31.877500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:31.886207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 38/72] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:31.887495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:31.916891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 35/73] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:31.916891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:31.924245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.013119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee824a39-6c7b-4686-86a4-5453e16e56f7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.037566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad8a09d-014c-41bd-8dc6-98043c520ddf None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.057630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.058467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.058884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.059332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.066204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.071167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:32.075674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.076310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:32.076408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:32.077345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:32.078115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:32.078685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.078685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.078800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 39/74] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:32.078800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.098533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.100199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.102028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 36/75] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:31 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.107512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.166794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.182944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879cb59-c5b9-4f7f-a804-fc0f422faba0 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.216570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.217226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.217511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.217789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.218368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f609cdbb-b819-418a-8a30-d880b059210d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.228053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:32.233327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:32.234139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:32.234877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 40/76] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:32.237673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.251873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.252573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.252950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.253289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.263656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:32.269736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:32.270638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:32.271400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 37/77] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:32.272040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.320614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.342294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c9ab0a-f8f4-491a-9129-3b5d2e2863c5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.374500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23721ebb-89a6-4525-b2b8-96c0e27a25b3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.374913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.375524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.375813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.376085 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.376744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:32.377078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.378467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.387211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 41/78] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.387998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.395934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.412553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.413194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.413483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.413762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.480118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.481547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.483962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b23a4-e818-4bb4-9e08-1da09d78faf6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.516818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.517460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.517740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.518012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.519115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.520527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.522096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 42/79] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.522857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.525198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.579827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 38/80] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:32.580714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.601734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.627030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f22c05b-4c64-4887-996a-06fff08e0cf5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.661365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.661972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.662259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.662539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.663207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:32.663515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.664964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.672706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 43/81] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.673521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.684370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab5cf7-e13c-403f-b049-d3201ac30720 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.718128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.718745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.719032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.719311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.719979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:32.720287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.721698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.730350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 39/82] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.733746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.743265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.769928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6759567d-c387-4e6c-8e82-357683d47855 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.803820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.804457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.804740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.805047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.806187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.807637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.809247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 44/83] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.810004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.815015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:32.841315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69df4b4-df0e-4dfc-87c8-ff820681f685 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.874557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.875338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.875621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.875892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.876528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.878011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.898508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 40/84] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:32 2022] POST /identity/v3/users => generated 289 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:32.913280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.913391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530153e0-a9db-4f9c-a414-eb5c67dbd526 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:32.953556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:32.954412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:32.954753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:32.955091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:32.956473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:32.958036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:32.959794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 45/85] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:32.960624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:32.980817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.027343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0e461a-ddf3-4d01-bf07-d3cc23a2e5c8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:33.041658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.067628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:33.068278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:33.068564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:33.068840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:33.070066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:33.071548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:33.077852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 41/86] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:33.077852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:33.080292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55caf399-0ab4-472b-87e5-6561b6ef5482 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:33.128306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:33.129028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:33.129312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:33.129584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:33.130708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:33.132155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:33.134291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 46/87] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:32 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:33.135129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:33.199044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fee1b7f-6255-4d3d-afac-e55486ee16ef None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:33.238365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.245028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:33.245924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:33.246422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:33.246873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:33.259480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70773bc-ffa8-43dd-b13c-2eb49e8ed118 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:33.313737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:33.314382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:33.314783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:33.315065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:33.316096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=b2a267a3bc0a40988be575ff351b6ef4) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:33.316413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:33.317947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:33.325650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 47/88] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:33 2022] GET /identity/v3/role_assignments?user.id=b2a267a3bc0a40988be575ff351b6ef4&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:33.326705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:33.331491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.343915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:33.345630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:33.478558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 42/89] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:33.479528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:33.516190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.531928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699129d0-5d30-409a-9b39-41b4c74d9852 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:33.616827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f058f5-cddd-457f-8c13-f94945c35867 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:33.617704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:33.619026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:33.619440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:33.619848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:33.682300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:33.682917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:33.683197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:33.683467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:33.740170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:33.741678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:33.745540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.815360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:33.816772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:33.873465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:33.931531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 48/90] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 605 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:33.932164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:34.002405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 43/91] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:34.003287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:34.128668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e1125-b8fa-4422-a4c0-8edacb4edf50 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:34.158283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:34.167369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:34.168022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:34.168306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:34.168606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:34.171185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:34.174224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:34.175893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 44/92] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:34 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:34.176680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:34.184067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88414cc-4ffb-4599-bda0-9b691228891e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:34.187894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:34.257955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:34.258650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:34.259449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:34.259783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:34.261365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:34.262837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:34.267312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 49/93] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:33 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:34.268117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:34.367691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a34434-f02c-4d43-9bf6-a7c3cd835e6e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:34.441029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:34.441663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:34.441953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:34.442230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] PATH_INFO: `/v3/projects/f8a6995874634badbfa912788f12780e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:34.443489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] RBAC: Authorizing `identity:get_project(project_id=f8a6995874634badbfa912788f12780e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:34.445963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:34.448212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 45/94] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:34 2022] GET /identity/v3/projects/f8a6995874634badbfa912788f12780e => generated 312 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:34.449010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:34.466552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d25dbc-b3f4-4559-84c0-a28074a73418 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:34.536448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:34.537090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:34.537378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:34.537658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:34.538388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] RBAC: Adding query filter params (role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=b2a267a3bc0a40988be575ff351b6ef4) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:34.538701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:34.540742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:34.562486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 50/95] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:34 2022] GET /identity/v3/role_assignments?user.id=b2a267a3bc0a40988be575ff351b6ef4&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 259 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:34.563316 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:34.569046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:34.605630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4c08d4-8f5d-402b-b177-ec27277f74f0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:34.653889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:34.676330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:34.676977 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:34.677262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:34.677537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:34.678279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=90cfae9be7734268a0ec1a4bc2e21b77) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:34.678597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:34.680048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f307fce-01ba-478d-b0fb-0851f4fef4d9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:34.693595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 46/96] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:34 2022] GET /identity/v3/role_assignments?user.id=90cfae9be7734268a0ec1a4bc2e21b77&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:34.767457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e900b257-0a36-4567-8cbd-71c4cbe5a923 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:34.816586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:34.817254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:34.817545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:34.817892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:34.818682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=f8a6995874634badbfa912788f12780e, user.id=b2a267a3bc0a40988be575ff351b6ef4) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:34.819052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:34.820627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4630c1-b09d-4bc8-9752-93db05c4a7e2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:34.827674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 51/97] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:34 2022] GET /identity/v3/role_assignments?user.id=b2a267a3bc0a40988be575ff351b6ef4&scope.project.id=f8a6995874634badbfa912788f12780e&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 259 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:35.440934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ba3300-f564-49f2-be3b-feb26d3351eb None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:35.441447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ba3300-f564-49f2-be3b-feb26d3351eb None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:35.441767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ba3300-f564-49f2-be3b-feb26d3351eb None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:35.443309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 47/98] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:35.449451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c286c-200e-4c37-81e9-acb7310ab58e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:35.449894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c286c-200e-4c37-81e9-acb7310ab58e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:35.449982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c286c-200e-4c37-81e9-acb7310ab58e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:35.478448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-119c286c-200e-4c37-81e9-acb7310ab58e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:35.479668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-119c286c-200e-4c37-81e9-acb7310ab58e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:35.574215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 52/99] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:35.952432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fe305f-9f47-4287-972a-9d2e08e3eac2 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:35.952802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fe305f-9f47-4287-972a-9d2e08e3eac2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:35.953016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fe305f-9f47-4287-972a-9d2e08e3eac2 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:35.954482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 48/100] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:35.963244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:35.963526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:35.963801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:35.994386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:35.995583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:36.020399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:36.038901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fe305f-9f47-4287-972a-9d2e08e3eac2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:36.078393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 53/101] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:36.208227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fe305f-9f47-4287-972a-9d2e08e3eac2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:36.241567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:36.243987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdebe70-0f1b-41ea-9242-e1cd70706192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:36.245092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdebe70-0f1b-41ea-9242-e1cd70706192 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:36.245764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdebe70-0f1b-41ea-9242-e1cd70706192 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:36.246414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdebe70-0f1b-41ea-9242-e1cd70706192 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:36.247950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdebe70-0f1b-41ea-9242-e1cd70706192 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:36.249452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdebe70-0f1b-41ea-9242-e1cd70706192 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:36.251032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 49/102] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:36.256609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:36.387804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7add2774-8143-4a00-b1dc-55e982b8d694 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:36.438205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386bc1c5-595a-4b2b-b8df-c8e52ca8c6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:36.439376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386bc1c5-595a-4b2b-b8df-c8e52ca8c6fc None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:36.439839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386bc1c5-595a-4b2b-b8df-c8e52ca8c6fc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:36.440297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386bc1c5-595a-4b2b-b8df-c8e52ca8c6fc None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:36.441191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386bc1c5-595a-4b2b-b8df-c8e52ca8c6fc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:36.442822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386bc1c5-595a-4b2b-b8df-c8e52ca8c6fc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:36.486329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 54/103] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:57:36 2022] POST /identity/v3/projects => generated 302 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:36.489041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442d581-61e1-48b8-b695-8ee47ab12184 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:36.489893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442d581-61e1-48b8-b695-8ee47ab12184 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:36.490393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b442d581-61e1-48b8-b695-8ee47ab12184 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:36.492024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 50/104] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:36.498519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:36.499128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:36.499598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:36.518937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:36.520639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:36.533874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:36.620888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 55/105] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:36.672634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b442d581-61e1-48b8-b695-8ee47ab12184 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:36.793641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b442d581-61e1-48b8-b695-8ee47ab12184 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:36.835877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:36.836533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:36.836921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:36.837197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:36.848903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:36.854584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:36.855431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:36.856185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 51/106] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:36.862025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:36.962407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:36.997761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1740a16e-6f33-423b-8cfc-0b5f8da3a829 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:37.049343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:37.065468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:37.066102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.066389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.066667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.067347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:37.067659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:37.069083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:37.087147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 56/107] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:37.093397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:37.100347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:37.133797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e145d-7cbf-402f-b5bc-02b08b79b9fb None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:37.204987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:37.205621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.205903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.206175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] PATH_INFO: `/v3/roles/member` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.222730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:37.229272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:37.230121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:37.230953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 52/108] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:57:37.239353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:37.289880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e333ae-b4d9-4a99-8111-dd58d1b1b63b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:37.363052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:37.363618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.363900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.364173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] PATH_INFO: `/v3/users/0146ee3f42ca414d92bbab06274620a6` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.393555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] RBAC: Authorizing `identity:get_user(user_id=0146ee3f42ca414d92bbab06274620a6)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:37.395025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:37.396913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 57/109] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:37 2022] GET /identity/v3/users/0146ee3f42ca414d92bbab06274620a6 => generated 289 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:37.403497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:37.419241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673d3713-9579-4a0d-b185-13bab1b985ac None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:37.455792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:37.494210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:37.495149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.495440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.495722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.496501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:37.496932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:37.498340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:37.517018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 53/110] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:37.523421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:37.528432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:37.594813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c229e56-cdae-4f0a-b345-cf14ea07a372 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:37.654770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c2a565-be4d-4daa-9a12-85047b5a038d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:37.655307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2a565-be4d-4daa-9a12-85047b5a038d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.655589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2a565-be4d-4daa-9a12-85047b5a038d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.655861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2a565-be4d-4daa-9a12-85047b5a038d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.677807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef192c-ef1d-4c48-b9e3-d20d782406dc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:37.754624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101ac705-2bf4-445f-809f-7be0486ed4ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:37.755744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101ac705-2bf4-445f-809f-7be0486ed4ff None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.756030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101ac705-2bf4-445f-809f-7be0486ed4ff None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.756310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101ac705-2bf4-445f-809f-7be0486ed4ff None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.757431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2a565-be4d-4daa-9a12-85047b5a038d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:37.757801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101ac705-2bf4-445f-809f-7be0486ed4ff None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:37.759761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101ac705-2bf4-445f-809f-7be0486ed4ff None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:37.759974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2a565-be4d-4daa-9a12-85047b5a038d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:37.761582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 54/111] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:37 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:37.762495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27310b89-876b-4152-95e8-00ad7d7a148f None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.762778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27310b89-876b-4152-95e8-00ad7d7a148f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.763051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27310b89-876b-4152-95e8-00ad7d7a148f None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.764404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 55/112] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:37.774557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27310b89-876b-4152-95e8-00ad7d7a148f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:37.827559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:37.867268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:37.875314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 58/113] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:37.878030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:37.878474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:37.879387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:37.903471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:37.904950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:37.980203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 59/114] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:37.981379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:38.013668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27310b89-876b-4152-95e8-00ad7d7a148f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:38.100582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb532fa1-22ad-4d50-bec9-9007c0004d46 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:38.114832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:38.140267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:38.141224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.141731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.142210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.143476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:38.145393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:38.146157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:38.146814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.147103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.147388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.147916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 60/115] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:37 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:38.153238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:38.153238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:38.153657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 56/116] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:37 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:38.154621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:38.159269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:38.319737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c6ea90-ec60-4254-8599-ac4ce631c91a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:38.330238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff37ecf-d1d2-4d77-ad4c-8a040c063a70 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:38.347196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:38.359948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:38.360556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.360840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.361144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.361868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] RBAC: Adding query filter params (role.id=bf23174ef65e4c6a817a0049740656f1, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=0146ee3f42ca414d92bbab06274620a6) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:38.362178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:38.363618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4623d27-7059-4b12-828b-9bcb0a5d7261 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:38.366134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d664e8e-fcb9-4ebc-a002-94f40077db09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:38.366793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d664e8e-fcb9-4ebc-a002-94f40077db09 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.367073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d664e8e-fcb9-4ebc-a002-94f40077db09 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.367346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d664e8e-fcb9-4ebc-a002-94f40077db09 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06/users/b2a267a3bc0a40988be575ff351b6ef4/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.369693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 61/117] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:38 2022] GET /identity/v3/role_assignments?user.id=0146ee3f42ca414d92bbab06274620a6&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=bf23174ef65e4c6a817a0049740656f1 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:38.370797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f85a95e-be33-4970-b67b-794438d8cd67 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.371090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f85a95e-be33-4970-b67b-794438d8cd67 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.371366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f85a95e-be33-4970-b67b-794438d8cd67 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.377111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 62/118] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:38.377111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7272a12-65ab-49c8-9def-c26f3b11c640 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.377111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7272a12-65ab-49c8-9def-c26f3b11c640 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.377111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7272a12-65ab-49c8-9def-c26f3b11c640 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.377111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 63/119] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:38.377111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7272a12-65ab-49c8-9def-c26f3b11c640 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:38.380683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d664e8e-fcb9-4ebc-a002-94f40077db09 None admin] RBAC: Authorizing `identity:create_grant(project_id=881f2ae0820241ec9d3216765931ba06, user_id=b2a267a3bc0a40988be575ff351b6ef4, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:38.382686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d664e8e-fcb9-4ebc-a002-94f40077db09 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:38.399317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 57/120] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:38 2022] PUT /identity/v3/projects/881f2ae0820241ec9d3216765931ba06/users/b2a267a3bc0a40988be575ff351b6ef4/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:57:38.401386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0fc54-5719-4712-aad6-5128119a9770 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.401927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0fc54-5719-4712-aad6-5128119a9770 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.402504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac0fc54-5719-4712-aad6-5128119a9770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.412515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:38.427443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5ac0fc54-5719-4712-aad6-5128119a9770 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:38.429152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5ac0fc54-5719-4712-aad6-5128119a9770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:38.511001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 58/121] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:38.518203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.518957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.519481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.521632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7272a12-65ab-49c8-9def-c26f3b11c640 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:38.551646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:38.553369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:38.605945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:38.605945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.605945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.605945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.633225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:38.639035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:38.639964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:38.640725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 64/122] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:38.650862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:38.709023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 59/123] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:38.803289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:38.874648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a17938-5402-4fcd-a8cb-3a678cabd7d5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:38.964002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:38.964551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:38.964831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:38.965129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:38.965810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:38.966116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:38.967489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:38.980117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 65/124] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:38.991228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.036603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.077571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:39.276268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcd87b2-8acf-46ed-8ba0-eda9b864ee09 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:39.287851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1deb37f-3934-4d23-a3ba-15359b4d3f18 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:39.349365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:39.350016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.350300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.350579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.352077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:39.354171 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:39.355771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 60/125] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:38 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:39.357711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:39.358360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.358715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.358993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.362286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.378866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:39.384440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:39.385301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:39.386054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:39.387454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 66/126] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:57:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:39.394765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.418714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:39.569139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a5148b-f0cf-4549-bbe1-ee2d2020d671 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:39.581206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:39.598008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff57a8-fec4-450c-a7fe-c5932a4def02 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:39.635662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:39.636304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.636588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.636862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.655056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:39.659900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:39.660734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:39.661530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 61/127] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:39.662163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.668292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:39.668964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.671567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.671852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.673118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:39.674673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:39.676462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 67/128] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:39 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:39.677103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.830889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:39.856892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde27832-e38c-425a-9c68-95fe4b87661e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:39.857181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6718b-036f-4d45-a2f9-ccb653d7dd32 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:39.891002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:39.891619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.891902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.892179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.892838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:39.893176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:39.894656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:39.896706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:39.897734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.898018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.898298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.898957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:39.899291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:39.901238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fea41e-443e-490d-a735-dd693bcd03ce None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:39.903811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 68/129] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:39.904717 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.909800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 62/130] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:57:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:39.910874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104cf83a-aeaa-414e-a995-3bd7ef86b667 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:39.911169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104cf83a-aeaa-414e-a995-3bd7ef86b667 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:39.911451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104cf83a-aeaa-414e-a995-3bd7ef86b667 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:39.912894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 63/131] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:39.918243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104cf83a-aeaa-414e-a995-3bd7ef86b667 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:39.966396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:40.014460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b5d73-2594-4052-aed2-5e7e964b4ce7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.038232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104cf83a-aeaa-414e-a995-3bd7ef86b667 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.049839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.050647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.050999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.051346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06/users/90cfae9be7734268a0ec1a4bc2e21b77/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.053357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] RBAC: Authorizing `identity:create_grant(project_id=881f2ae0820241ec9d3216765931ba06, user_id=90cfae9be7734268a0ec1a4bc2e21b77, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.054874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.059314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:40.077339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 69/132] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:39 2022] PUT /identity/v3/projects/881f2ae0820241ec9d3216765931ba06/users/90cfae9be7734268a0ec1a4bc2e21b77/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:57:40.078267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:40.098340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbdd518-4939-4115-ad0c-ae0fb4102354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.098963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbdd518-4939-4115-ad0c-ae0fb4102354 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.099257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbdd518-4939-4115-ad0c-ae0fb4102354 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.100289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbdd518-4939-4115-ad0c-ae0fb4102354 None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.104577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbdd518-4939-4115-ad0c-ae0fb4102354 None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.109437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbdd518-4939-4115-ad0c-ae0fb4102354 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.113967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 64/133] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:39 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:40.118665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.118975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.119215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.161949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:40.166630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:40.289577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93b3650-1870-4ce4-b304-f657fff5905f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.314749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 65/134] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:40.315754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:40.332073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.332670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.332986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.333264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.337908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.339435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.341269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 70/135] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:40 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:40.343096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:40.347268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:40.412664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:40.453719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8e6794-de0b-4af4-8d99-da5b8b52cdf1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.482842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23cf87c-591c-4bc4-948e-f39e22c028e9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.489691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.490335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.490622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.490900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.492024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.493649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.495353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 71/136] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:40 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:40.505886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:40.613596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.614244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.614532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.614813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] PATH_INFO: `/v3/projects/f8a6995874634badbfa912788f12780e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.616964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] RBAC: Authorizing `identity:get_project(project_id=f8a6995874634badbfa912788f12780e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.618492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.620459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 66/137] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:40 2022] GET /identity/v3/projects/f8a6995874634badbfa912788f12780e => generated 312 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:40.629798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:40.667625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bba72d7-1dbb-407f-8420-608a57831ae3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.745235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:40.767005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.767005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.767257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.767500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06/users/b2a267a3bc0a40988be575ff351b6ef4/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.774915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] RBAC: Authorizing `identity:create_grant(project_id=881f2ae0820241ec9d3216765931ba06, user_id=b2a267a3bc0a40988be575ff351b6ef4, role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.776414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.810352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 72/138] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:40 2022] PUT /identity/v3/projects/881f2ae0820241ec9d3216765931ba06/users/b2a267a3bc0a40988be575ff351b6ef4/roles/609c7d98c34d4dba86ff458ea0726951 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:57:40.811284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:40.831931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:40.843329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e500b927-619e-47b7-8170-8b71917d24a9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.902797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.903615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.903903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.904234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] PATH_INFO: `/v3/projects/f8a6995874634badbfa912788f12780e/users/b2a267a3bc0a40988be575ff351b6ef4/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:40.908404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] RBAC: Authorizing `identity:create_grant(project_id=f8a6995874634badbfa912788f12780e, user_id=b2a267a3bc0a40988be575ff351b6ef4, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:40.910175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:40.943789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 67/139] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:40 2022] PUT /identity/v3/projects/f8a6995874634badbfa912788f12780e/users/b2a267a3bc0a40988be575ff351b6ef4/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:57:40.959941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa6bbdf-901a-490f-ab18-4fa57a3e61e4 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:40.998716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff797e3f-4027-4ee2-acc4-258b7b91fed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:40.999377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff797e3f-4027-4ee2-acc4-258b7b91fed0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:40.999657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff797e3f-4027-4ee2-acc4-258b7b91fed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:40.999928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff797e3f-4027-4ee2-acc4-258b7b91fed0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:41.001768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff797e3f-4027-4ee2-acc4-258b7b91fed0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:41.003173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff797e3f-4027-4ee2-acc4-258b7b91fed0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:41.005012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 73/140] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:41.010897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:41.155867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75338fca-62c1-4c9e-add8-10b597667ee5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:41.194497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39ddb98-daf8-4ab9-a118-8731f9b2c9dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:41.195044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39ddb98-daf8-4ab9-a118-8731f9b2c9dc None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:41.195331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39ddb98-daf8-4ab9-a118-8731f9b2c9dc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:41.195611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39ddb98-daf8-4ab9-a118-8731f9b2c9dc None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:41.196857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39ddb98-daf8-4ab9-a118-8731f9b2c9dc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:41.198364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39ddb98-daf8-4ab9-a118-8731f9b2c9dc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:41.222267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 68/141] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:41 2022] POST /identity/v3/users => generated 283 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:41.408597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9d3d7-f831-49fd-aa27-b4aa32714047 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:41.408975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9d3d7-f831-49fd-aa27-b4aa32714047 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:41.409170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9d3d7-f831-49fd-aa27-b4aa32714047 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:41.410812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 74/142] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:41.416602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:41.416912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:41.417182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:41.440178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:41.441406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:41.443549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:41.474326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348323dd-0b77-4b3a-a40c-90fdc2ccd430 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:41.475177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348323dd-0b77-4b3a-a40c-90fdc2ccd430 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:41.475606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348323dd-0b77-4b3a-a40c-90fdc2ccd430 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:41.476970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 75/143] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:41.483056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:41.483342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:41.483626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:41.518187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:41.519407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:41.532380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 69/144] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:41.595800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:41.611611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 76/145] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:41.943188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:41.982370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.076644 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f403255-7f5a-4d4d-a43f-bedacb882b53 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.127471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3544d8be-04cc-4800-a338-9d8f80634566 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.156320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.156996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.157279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.157552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.170259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.171187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.171639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.172108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.175270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:42.183421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:42.194552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:42.195429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:42.196702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:42.198391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:42.203440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:42.204485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:42.205485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:42.207507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 70/146] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:42.213487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 77/147] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:57:42.214974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.219400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.320757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:42.407855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0dfb01-3c99-4abe-b383-10eb48fb991d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.423190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a726bd6-ccea-4d1a-a64d-cc7fcba5cde9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.476010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.476686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.477049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.477439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.478302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:42.478668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:42.480127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:42.495851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 78/148] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:42.501538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.502154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.503049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.503541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.504032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.504970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:42.505493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:42.507308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:42.522385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 71/149] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:42.529116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.619817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:42.665208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7e1c04-596d-4e3a-8328-bcebffa6b240 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.672741 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be093e0-f796-4e8e-be4b-b37e2a949c48 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.687021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:42.715524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.716265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.716369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.716553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.716804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.716994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.717281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.717558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] PATH_INFO: `/v3/users/0146ee3f42ca414d92bbab06274620a6` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.718741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:42.720202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:42.721563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] RBAC: Authorizing `identity:get_user(user_id=0146ee3f42ca414d92bbab06274620a6)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:42.722002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 79/150] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:42 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:42.723163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:42.725061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 72/151] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:42 2022] GET /identity/v3/users/0146ee3f42ca414d92bbab06274620a6 => generated 289 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:42.728009 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.730814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:42.867575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1675ca-4fc2-4f8c-8343-ad479e981208 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.905699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:42.922534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:42.923187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:42.923469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:42.923725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91057e65-7dca-4b72-a1e4-6d830e9f47eb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:42.923827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:42.929557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:42.931085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:42.933135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 73/152] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:42 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:42.939061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:43.013623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eafcfb-d426-4383-adf3-5dc64488eade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:43.014343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eafcfb-d426-4383-adf3-5dc64488eade None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:43.014526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eafcfb-d426-4383-adf3-5dc64488eade None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:43.014767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eafcfb-d426-4383-adf3-5dc64488eade None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:43.076669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112569d5-ebe2-4f5b-90a1-122b7508f161 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:43.118037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:43.134689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:43.135461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:43.135736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:43.136015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06/users/0146ee3f42ca414d92bbab06274620a6/roles/bf23174ef65e4c6a817a0049740656f1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:43.142291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] RBAC: Authorizing `identity:create_grant(project_id=881f2ae0820241ec9d3216765931ba06, user_id=0146ee3f42ca414d92bbab06274620a6, role_id=bf23174ef65e4c6a817a0049740656f1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:43.143780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:43.147252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eafcfb-d426-4383-adf3-5dc64488eade None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:43.150909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eafcfb-d426-4383-adf3-5dc64488eade None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:43.171155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 74/153] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:42 2022] PUT /identity/v3/projects/881f2ae0820241ec9d3216765931ba06/users/0146ee3f42ca414d92bbab06274620a6/roles/bf23174ef65e4c6a817a0049740656f1 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:57:43.309930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 80/154] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 582 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:57:43.316218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:43.390470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb97c5-60fe-486d-a373-39fe8583fa2c None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:43.391099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb97c5-60fe-486d-a373-39fe8583fa2c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:43.391580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb97c5-60fe-486d-a373-39fe8583fa2c None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:43.395940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 81/155] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:43.402224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:43.402798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:43.403289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:43.414299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:43.441959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:43.443548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:43.487040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3419a71c-1ea1-4ce3-932b-deb7e464fcf8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:43.570291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:43.570291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:43.570291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:43.570291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:43.570928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:43.572910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:43.579253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 75/156] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:43 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:43.583933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:43.608121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 82/157] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:43.719993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6134fce8-f07c-433e-8b02-67d6b423fda9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:43.745744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:43.757923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:43.758534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:43.759341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:43.759614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:43.760340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=b2a267a3bc0a40988be575ff351b6ef4) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:43.760646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:43.762107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:43.767874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 76/158] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:43 2022] GET /identity/v3/role_assignments?user.id=b2a267a3bc0a40988be575ff351b6ef4&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:44.149162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:44.161453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:44.264530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3d9d61-3fc5-4c14-a988-a2a608252530 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:44.301420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:44.302056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:44.302333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:44.302601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:44.319177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:44.324554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:44.325475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:44.326309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 83/159] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:44.332985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:44.469498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e89a0-10c4-4271-9d72-6955aaafb302 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:44.506176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:44.506799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:44.507078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:44.507352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:44.508041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:44.508325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:44.510282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:44.521161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 77/160] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:44.526988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:44.621714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:44.657783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7859976-3eab-4d67-beb1-a0f5f111ce43 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:44.698257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:44.699239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:44.699732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:44.700190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] PATH_INFO: `/v3/users/90cfae9be7734268a0ec1a4bc2e21b77` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:44.701590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] RBAC: Authorizing `identity:get_user(user_id=90cfae9be7734268a0ec1a4bc2e21b77)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:44.703232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:44.705341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 84/161] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:44 2022] GET /identity/v3/users/90cfae9be7734268a0ec1a4bc2e21b77 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:44.712946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:44.813115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:44.867220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c0bbd9-5310-4c8e-9718-c4e979ece076 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:44.904315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65f1d36-e488-4758-b5db-181f2d884821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:44.905014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65f1d36-e488-4758-b5db-181f2d884821 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:44.905298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65f1d36-e488-4758-b5db-181f2d884821 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:44.905570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65f1d36-e488-4758-b5db-181f2d884821 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:44.981451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65f1d36-e488-4758-b5db-181f2d884821 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:44.982918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65f1d36-e488-4758-b5db-181f2d884821 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:45.037020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:45.111022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 78/162] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:45.117596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:45.211772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a255a4af-1fa5-4934-baad-f8abb25da849 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.212181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a255a4af-1fa5-4934-baad-f8abb25da849 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.212396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a255a4af-1fa5-4934-baad-f8abb25da849 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.214043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 79/163] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:45.218917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b721bc-61a9-469e-812d-4a0c28620db8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.219200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b721bc-61a9-469e-812d-4a0c28620db8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.219476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b721bc-61a9-469e-812d-4a0c28620db8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.247576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a6b721bc-61a9-469e-812d-4a0c28620db8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:45.248985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a6b721bc-61a9-469e-812d-4a0c28620db8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:45.270882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b589ed-4ecb-4210-9ac3-d829dc3396c0 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:45.329649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:45.330295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.330571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.330837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.333328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:45.334787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:45.336408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 85/164] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:45 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:45.343472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:45.352325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:45.379106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 80/165] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:45.440369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dda50-1c52-44f3-a4b2-2a0775e5e6f5 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.440686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dda50-1c52-44f3-a4b2-2a0775e5e6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.440953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90dda50-1c52-44f3-a4b2-2a0775e5e6f5 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.442293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 81/166] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:45.468307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486127c-3eca-4978-a1e4-88a90b52d124 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.468603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486127c-3eca-4978-a1e4-88a90b52d124 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.468849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486127c-3eca-4978-a1e4-88a90b52d124 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.486979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc8d2b9-a568-4d30-801c-b8caa05fcf4d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:45.500382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9486127c-3eca-4978-a1e4-88a90b52d124 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:45.501678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9486127c-3eca-4978-a1e4-88a90b52d124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:45.542522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:45.543134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.543411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.543678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.544394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=90cfae9be7734268a0ec1a4bc2e21b77) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:45.544696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:45.546485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:45.556455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 86/167] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:45 2022] GET /identity/v3/role_assignments?user.id=90cfae9be7734268a0ec1a4bc2e21b77&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:45.608725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 82/168] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:45.825285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:45.838968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2f583-0446-43b8-b823-d736674748c6 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.839616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2f583-0446-43b8-b823-d736674748c6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.840082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2f583-0446-43b8-b823-d736674748c6 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.842078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 83/169] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:45.848654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:45.849258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:45.849884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:45.872048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:45.873882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:45.878557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:45.951358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:46.005905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709d41c-598b-42e0-8fa9-aace0de4c16c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.019180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 84/170] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:46.064564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.065227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.065509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.065778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.082883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:46.088314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:46.089271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:46.090061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:46.092983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 87/171] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:46.095127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.122574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.278153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7f3ca6-c942-491f-8a55-8442a1b5eeee None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.307437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def3748a-44ad-42ff-8383-b18188c58a5a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.345077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.345611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.345886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.346154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.346828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:46.347127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:46.348503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:46.361554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 85/172] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:46.364188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:46.374340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.374979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.375257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.375529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.378097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.388154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:46.403745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:46.409035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:46.409977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:46.410766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:46.411816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 88/173] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:57:46.422549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.583829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cce2ed-c1da-4c96-9ecf-af6c35430883 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.608292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75162961-f1b6-4667-b56d-9286bfe11baa None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.625830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.626452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.626727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.627053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.627724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:46.628739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:46.630156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:46.644402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 89/174] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:46.645238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.675856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.676591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.676884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.677206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.679124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:46.680581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:46.682226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 86/175] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:46.683819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.744350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:46.804727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc045f1-c987-4f02-b789-094770e34090 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.808987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:46.845965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.846596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.846870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.847139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.855044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c1f46f-d9c4-4f9f-a2d7-849ed84110f4 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.860793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:46.866779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:46.867655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:46.869042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 90/176] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:46.869709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.904289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e53cfa-433f-4a68-b058-cba3badb8512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:46.905296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e53cfa-433f-4a68-b058-cba3badb8512 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.905585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e53cfa-433f-4a68-b058-cba3badb8512 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.906001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e53cfa-433f-4a68-b058-cba3badb8512 None admin] PATH_INFO: `/v3/users/b2a267a3bc0a40988be575ff351b6ef4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.907325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e53cfa-433f-4a68-b058-cba3badb8512 None admin] RBAC: Authorizing `identity:get_user(user_id=b2a267a3bc0a40988be575ff351b6ef4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:46.909172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e53cfa-433f-4a68-b058-cba3badb8512 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:46.910872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 87/177] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/users/b2a267a3bc0a40988be575ff351b6ef4 => generated 275 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:46.911935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf4f15-877f-478a-85d9-2f45c71b903f None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:46.912154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf4f15-877f-478a-85d9-2f45c71b903f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:46.912430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf4f15-877f-478a-85d9-2f45c71b903f None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:46.913774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 88/178] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:46 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:46.916986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf4f15-877f-478a-85d9-2f45c71b903f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:46.982685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b96862-a9ed-4196-9a66-20aa82f660e9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:46.995746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.016975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe70c5-fb45-429b-8412-16417aee8138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.017159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf4f15-877f-478a-85d9-2f45c71b903f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.017623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe70c5-fb45-429b-8412-16417aee8138 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.017909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe70c5-fb45-429b-8412-16417aee8138 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.018184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe70c5-fb45-429b-8412-16417aee8138 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.049220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.049824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.050109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.050393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.051041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:47.051351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.052720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668bf8a0-5703-4cfc-9ed5-4485cf0224b9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.058144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.066906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 89/179] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:47.068147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373db575-ce05-4ad0-8d74-bedd4b352dfc None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.068434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373db575-ce05-4ad0-8d74-bedd4b352dfc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.068710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373db575-ce05-4ad0-8d74-bedd4b352dfc None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.069980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 90/180] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:47.070691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373db575-ce05-4ad0-8d74-bedd4b352dfc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.085727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe70c5-fb45-429b-8412-16417aee8138 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.087746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe70c5-fb45-429b-8412-16417aee8138 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.174106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373db575-ce05-4ad0-8d74-bedd4b352dfc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.186167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 91/181] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:47.186167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a38ffe-528f-4203-ad7b-d7edbe20adbf None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.186167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a38ffe-528f-4203-ad7b-d7edbe20adbf None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.186167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a38ffe-528f-4203-ad7b-d7edbe20adbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.205773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c7a38ffe-528f-4203-ad7b-d7edbe20adbf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:47.206971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c7a38ffe-528f-4203-ad7b-d7edbe20adbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:47.212483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a785b352-c3cf-46eb-9ffb-ef700987b309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.213117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a785b352-c3cf-46eb-9ffb-ef700987b309 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.213398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a785b352-c3cf-46eb-9ffb-ef700987b309 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.213668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a785b352-c3cf-46eb-9ffb-ef700987b309 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.214026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.273608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 92/182] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:47.275157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811b3010-7488-4bee-822f-6f86ef158909 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.275445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811b3010-7488-4bee-822f-6f86ef158909 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.275714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811b3010-7488-4bee-822f-6f86ef158909 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.276984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 93/183] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:47.281637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811b3010-7488-4bee-822f-6f86ef158909 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.287757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.291109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a785b352-c3cf-46eb-9ffb-ef700987b309 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.292263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a785b352-c3cf-46eb-9ffb-ef700987b309 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.392541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 91/184] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:47.394449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6efcc7-79c3-46b6-b609-45411e0c7ae9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.394944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6efcc7-79c3-46b6-b609-45411e0c7ae9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.395353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6efcc7-79c3-46b6-b609-45411e0c7ae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.398114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811b3010-7488-4bee-822f-6f86ef158909 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.415366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9f6efcc7-79c3-46b6-b609-45411e0c7ae9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:47.416743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9f6efcc7-79c3-46b6-b609-45411e0c7ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:47.433373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.433937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.434212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.434480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] PATH_INFO: `/v3/users/44cef300a7544102a2460ce978c5c9ed` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.445693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.449174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] RBAC: Authorizing `identity:get_user(user_id=44cef300a7544102a2460ce978c5c9ed)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.452077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.453949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 94/185] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/users/44cef300a7544102a2460ce978c5c9ed => generated 283 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:47.454733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.497756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 92/186] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:47.499216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.499738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.500187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.524214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:47.526110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:47.554950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.574466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138c97fa-49d9-40f8-9478-efd3cc8a2f27 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.604765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 93/187] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:47.605473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.611474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.612111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.612392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.612671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] PATH_INFO: `/v3/projects/f8a6995874634badbfa912788f12780e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.613856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] RBAC: Authorizing `identity:get_project(project_id=f8a6995874634badbfa912788f12780e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.615315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.617228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 95/188] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/projects/f8a6995874634badbfa912788f12780e => generated 312 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:47.617999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.703496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.715212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18933656-975f-40b3-891f-b1f1e9a6b567 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.744809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c780e0-3e37-47db-ab35-05baa5420193 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.752834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.753771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.754248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.754781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.756170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.757910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.760322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 94/189] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:47.761388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.783794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.784405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.784687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.784990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.792973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.855961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:47.857408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:47.882053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c872ba-cabe-4d1d-8c27-de1570a08007 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:47.919256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:47.919869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:47.920196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:47.920474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:47.921578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:47.931248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:47.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:47.936945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 95/190] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:47.937812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:47.963902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 96/191] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:47.963902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.027370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.039486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8f4256-519e-40c3-9cd9-6bd61b8737d9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.054601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49521c5e-7ab9-472f-8a23-9fe9f13447bb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.073136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.073744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.074024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.074718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.075451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=f8a6995874634badbfa912788f12780e, user.id=b2a267a3bc0a40988be575ff351b6ef4) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:48.075761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.077255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.082980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 96/192] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/role_assignments?user.id=b2a267a3bc0a40988be575ff351b6ef4&scope.project.id=f8a6995874634badbfa912788f12780e&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.083759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.087705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.088434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.088710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.089021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.089755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] RBAC: Adding query filter params (role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=b2a267a3bc0a40988be575ff351b6ef4) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:48.090059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.091550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.104823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 97/193] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:47 2022] GET /identity/v3/role_assignments?user.id=b2a267a3bc0a40988be575ff351b6ef4&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.104823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.172807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.215703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e872ef73-039b-448b-901b-4dc829aab9e2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.260355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.261154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.261509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.261873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.276759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:48.285379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:48.286335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:48.287129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 97/194] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:48.287772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6030e2-e925-4a41-be56-533955176996 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.287842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.308206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.342212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.343601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.343882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.344149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.346117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.346952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.348915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 98/195] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.349668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.412421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e53fe33-1da3-4692-8348-86c9070b77a5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.450146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b413d700-b193-480f-9588-236e932f3dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.450755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b413d700-b193-480f-9588-236e932f3dde None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.451037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b413d700-b193-480f-9588-236e932f3dde None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.451313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b413d700-b193-480f-9588-236e932f3dde None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.451975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b413d700-b193-480f-9588-236e932f3dde None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:48.452268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b413d700-b193-480f-9588-236e932f3dde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.453675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b413d700-b193-480f-9588-236e932f3dde None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.455358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.458281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1267b3b7-4232-4917-a4cd-78389327c98e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.461874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 98/196] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.462858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b413d700-b193-480f-9588-236e932f3dde None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.492605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.493737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.494017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.494288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.495422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.496897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.498516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 99/197] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.499259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.540779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.568256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b413d700-b193-480f-9588-236e932f3dde None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.603844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.604451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.604734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.605071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.605738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:48.606043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.606702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c377cff-f194-46ba-8485-db71d534d647 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.607430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.615985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 99/198] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.616791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.643158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.643766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.644047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.644324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.645002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.646459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.673083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 100/199] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:48 2022] POST /identity/v3/users => generated 297 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:48.677883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.714698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.725226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f2fe2-aded-4321-b2d3-a3c389a3035c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.765377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.766032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.766321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.766589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] PATH_INFO: `/v3/users/44cef300a7544102a2460ce978c5c9ed` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.767835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] RBAC: Authorizing `identity:get_user(user_id=44cef300a7544102a2460ce978c5c9ed)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.769348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.775610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 100/200] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/users/44cef300a7544102a2460ce978c5c9ed => generated 283 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.775610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.779606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.800882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fda476-1b15-405f-87ac-72dc798501eb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.836532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.837297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.837531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.837809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.838551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=44cef300a7544102a2460ce978c5c9ed) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:48.838864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.840327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.846843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 101/201] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/role_assignments?user.id=44cef300a7544102a2460ce978c5c9ed&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.854129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.903176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2eee5f-1ffa-4d50-b21d-852c2de23e73 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:48.958613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:48.959386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:48.959797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:48.960179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] PATH_INFO: `/v3/users/0146ee3f42ca414d92bbab06274620a6` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:48.962011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] RBAC: Authorizing `identity:get_user(user_id=0146ee3f42ca414d92bbab06274620a6)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:48.963703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:48.965643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 101/202] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/users/0146ee3f42ca414d92bbab06274620a6 => generated 289 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:48.971590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:48.972441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:48.989747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9458b6e6-a307-46c2-a7d2-b29c3f7921f8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:49.031445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:49.032003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:49.032674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:49.032996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:49.080616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:49.123887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e553e3-4a81-4199-a749-1f3a0c95f5c7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:49.129576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:49.131824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:49.171135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:49.171877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:49.172158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:49.172430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:49.231187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:49.246712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 102/203] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:49.252512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:49.260449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:49.263223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:49.349525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:49.373175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3333d89-7a97-4dcd-a27e-05cbe3ef5acb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:49.411078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:49.411718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:49.412000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:49.412272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:49.413629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:49.415309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:49.416063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 102/204] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:49.417373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 103/205] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:49 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:49.424589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:49.426103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:49.529887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:49.568615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf54ae48-3d81-427b-a3ff-7ab8a4032c1d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:49.613701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:49.614238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:49.614514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:49.614784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:49.615512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] RBAC: Adding query filter params (role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=44cef300a7544102a2460ce978c5c9ed) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:49.615822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:49.617279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:49.625294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566b63e3-6b9e-4ddb-9082-aff2c9257949 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:49.633077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 103/206] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:49 2022] GET /identity/v3/role_assignments?user.id=44cef300a7544102a2460ce978c5c9ed&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:49.667858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6ffb65-9186-4264-bf1b-94ad894378ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:49.668567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6ffb65-9186-4264-bf1b-94ad894378ef None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:49.668849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6ffb65-9186-4264-bf1b-94ad894378ef None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:49.669196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6ffb65-9186-4264-bf1b-94ad894378ef None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:49.670396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6ffb65-9186-4264-bf1b-94ad894378ef None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:49.671853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6ffb65-9186-4264-bf1b-94ad894378ef None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:49.673545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 104/207] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:49 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:49.679887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:49.719734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:49.794293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c6256f-2efa-497f-93d7-432f8acf4d0e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:49.831833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:49.832453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:49.832733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:49.833040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:49.833785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] RBAC: Adding query filter params (role.id=bf23174ef65e4c6a817a0049740656f1, scope.project.id=881f2ae0820241ec9d3216765931ba06, user.id=0146ee3f42ca414d92bbab06274620a6) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:49.834096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:49.835560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44444850-988a-468b-acc4-68af9671d1b0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:49.849059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 104/208] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:49 2022] GET /identity/v3/role_assignments?user.id=0146ee3f42ca414d92bbab06274620a6&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=bf23174ef65e4c6a817a0049740656f1 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:51.217162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ac4f3d-77cc-4c70-81df-ae305a5243a3 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.217708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ac4f3d-77cc-4c70-81df-ae305a5243a3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.217708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ac4f3d-77cc-4c70-81df-ae305a5243a3 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.219026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 105/209] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:51.224129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56628096-a7e4-4ee2-8e82-f3ac8fcb8e10 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.224527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56628096-a7e4-4ee2-8e82-f3ac8fcb8e10 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.224702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56628096-a7e4-4ee2-8e82-f3ac8fcb8e10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.244042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-56628096-a7e4-4ee2-8e82-f3ac8fcb8e10 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:51.245285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-56628096-a7e4-4ee2-8e82-f3ac8fcb8e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:51.259885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd0e282-7c9b-441b-ad85-b466e8f9f4ea None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.260295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd0e282-7c9b-441b-ad85-b466e8f9f4ea None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.260664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd0e282-7c9b-441b-ad85-b466e8f9f4ea None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.262360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 106/210] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:57:51.267211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11993743-483e-4468-8471-cbf3518bd359 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.267570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11993743-483e-4468-8471-cbf3518bd359 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.267911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11993743-483e-4468-8471-cbf3518bd359 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.285848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-11993743-483e-4468-8471-cbf3518bd359 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:51.287119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-11993743-483e-4468-8471-cbf3518bd359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:51.319073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 105/211] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:51.355290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 107/212] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:51.527073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f8562-88a1-43ca-a25d-e62dd8a945cd None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.527689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f8562-88a1-43ca-a25d-e62dd8a945cd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.528218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f8562-88a1-43ca-a25d-e62dd8a945cd None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.530351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 106/213] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:51.536748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57076b5b-4a42-4205-924c-97993d393820 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.537127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57076b5b-4a42-4205-924c-97993d393820 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.537401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57076b5b-4a42-4205-924c-97993d393820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.555400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-57076b5b-4a42-4205-924c-97993d393820 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:51.557133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-57076b5b-4a42-4205-924c-97993d393820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:51.600080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f8562-88a1-43ca-a25d-e62dd8a945cd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:51.612024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:51.630252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 108/214] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:51.635921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:51.636046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57076b5b-4a42-4205-924c-97993d393820 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:51.721563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f8562-88a1-43ca-a25d-e62dd8a945cd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:51.757944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57076b5b-4a42-4205-924c-97993d393820 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:51.766997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:51.767820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.768235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.768636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.779796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:51.784803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:51.785830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:51.786625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:51.788818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 107/215] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:51.795439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:51.803930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:51.805359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.805920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.806398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.822026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:51.829408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:51.830274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:51.831051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 109/216] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:57:51.837660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:51.885226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:51.917813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:51.927830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eb60c0-4777-4555-ac6f-b64eb4f5fd46 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:51.943917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c758751-d0e2-41ca-8bf2-e79e41f57e6c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:51.965930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:51.966553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.966831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.967272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.967764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:51.968061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:51.969516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:51.984651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 108/217] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:51.984651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:51.987958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:51.988576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:51.988854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:51.989149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:51.989818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:51.990445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:51.991595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.000917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 110/218] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:52.000917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.094056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636178c7-17be-400a-9e14-abdb9a81d818 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.105557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e502968-32e4-4584-b2bd-a00e121591b3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.112534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.127251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.127868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.128142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.128414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.129514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.130942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.132499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 109/219] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:52.133272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.140255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbccc80-ddc4-457b-9715-608ee8ec9224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.140907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbccc80-ddc4-457b-9715-608ee8ec9224 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.141188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbccc80-ddc4-457b-9715-608ee8ec9224 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.141455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbccc80-ddc4-457b-9715-608ee8ec9224 None admin] PATH_INFO: `/v3/users/44cef300a7544102a2460ce978c5c9ed` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.142580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbccc80-ddc4-457b-9715-608ee8ec9224 None admin] RBAC: Authorizing `identity:get_user(user_id=44cef300a7544102a2460ce978c5c9ed)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.144020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbccc80-ddc4-457b-9715-608ee8ec9224 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.145607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 111/220] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/users/44cef300a7544102a2460ce978c5c9ed => generated 283 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:52.146686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b3a16-1540-4e82-9504-a12257819dbd None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.146966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b3a16-1540-4e82-9504-a12257819dbd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.147233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b3a16-1540-4e82-9504-a12257819dbd None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.148407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 112/221] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:52.149104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b3a16-1540-4e82-9504-a12257819dbd None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.159991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.239744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1736784e-0562-46ee-97bd-a98fe4b22b8b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.250425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b3a16-1540-4e82-9504-a12257819dbd None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.271296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.271919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.272196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.272468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] PATH_INFO: `/v3/users/e53c94d0230b46879f98f2f0fe8d34ca` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.281796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f50a3d3-1e17-49f8-9e0e-feaa153e99a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.282395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f50a3d3-1e17-49f8-9e0e-feaa153e99a7 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.282668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f50a3d3-1e17-49f8-9e0e-feaa153e99a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.282935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f50a3d3-1e17-49f8-9e0e-feaa153e99a7 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.283576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f50a3d3-1e17-49f8-9e0e-feaa153e99a7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.285028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f50a3d3-1e17-49f8-9e0e-feaa153e99a7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.286151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] RBAC: Authorizing `identity:get_user(user_id=e53c94d0230b46879f98f2f0fe8d34ca)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.287587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.289230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 110/222] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/users/e53c94d0230b46879f98f2f0fe8d34ca => generated 297 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:52.289992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.306663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 113/223] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:52 2022] POST /identity/v3/users => generated 297 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:52.307865 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.308149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.308419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.330834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:52.333846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:52.363623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.388001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.415250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd3071d-d50c-43f5-ac7f-eb0c5f12eee5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.416074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 114/224] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:52.416764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.477532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.478732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.479640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.481089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.483198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.483793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.488934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 111/225] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:52.492252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.547263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04fb2f-154b-4d71-aa76-2cd74ef000ef None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.594264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.594264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.594264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.596506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.623121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e6c1db-47b8-469d-924e-193918622516 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.632614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.674269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.675267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.675562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.675839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/44cef300a7544102a2460ce978c5c9ed/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.677927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] RBAC: Authorizing `identity:create_grant(project_id=a7010d3512da4f9c9fab7d993427ebf8, user_id=44cef300a7544102a2460ce978c5c9ed, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.679387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.681059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:52.683137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:52.696026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 112/226] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:52 2022] PUT /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/44cef300a7544102a2460ce978c5c9ed/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:57:52.705394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.720158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.829441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 115/227] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:52.836924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.908052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390a00d-bc19-4602-87db-5f8ebe1dd1ec None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.908918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:52.946246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.946888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.947166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.947441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:52.958189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:52.960467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6767608-5917-47fa-8bd1-91c17e353188 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:52.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:52.964536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:52.965362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 113/228] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:52.973876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:52.998491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:52.999126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:52.999448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:52.999778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:53.001012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:53.002670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:53.004547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 116/229] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:53.010674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:53.054807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:53.102979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651e2c42-935c-4278-81fe-53aa0a7086fb None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:53.152920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:53.152920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:53.152920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:53.152920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:53.153373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:53.153373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:53.153373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:53.157778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1001634d-60f3-44da-8d51-9565912cad8b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:53.169636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 114/230] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:57:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:53.169636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:53.210674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:53.224315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:53.224956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:53.225239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:53.225512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:53.226300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=e53c94d0230b46879f98f2f0fe8d34ca) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:53.226610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:53.228240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:53.234154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 117/231] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:53 2022] GET /identity/v3/role_assignments?user.id=e53c94d0230b46879f98f2f0fe8d34ca&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 259 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:53.287260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f59bd7b-14a4-4940-93be-32a83bead437 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:53.317413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:53.326325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:53.326961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:53.327236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:53.327510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] PATH_INFO: `/v3/users/44cef300a7544102a2460ce978c5c9ed` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:53.328835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] RBAC: Authorizing `identity:get_user(user_id=44cef300a7544102a2460ce978c5c9ed)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:53.330330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:53.332287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 115/232] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:53 2022] GET /identity/v3/users/44cef300a7544102a2460ce978c5c9ed => generated 283 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:53.338041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:53.456275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad3e92-05c2-474d-90bf-46be822c480a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:53.493328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fd978f-e196-409d-8cbc-fe5e0aae09f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:53.493966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd978f-e196-409d-8cbc-fe5e0aae09f0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:53.494294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd978f-e196-409d-8cbc-fe5e0aae09f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:53.494626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd978f-e196-409d-8cbc-fe5e0aae09f0 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:53.495797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fd978f-e196-409d-8cbc-fe5e0aae09f0 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:53.497356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fd978f-e196-409d-8cbc-fe5e0aae09f0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:53.499269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 118/233] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:53 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:53.505176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:53.612086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b312447e-24ac-4024-bd79-cb8d8ccc579e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:53.646598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f7f88-65db-4082-8ca1-9097f44048bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:53.647255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f7f88-65db-4082-8ca1-9097f44048bf None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:53.647573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f7f88-65db-4082-8ca1-9097f44048bf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:53.647854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f7f88-65db-4082-8ca1-9097f44048bf None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/44cef300a7544102a2460ce978c5c9ed/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:53.650526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18f7f88-65db-4082-8ca1-9097f44048bf None admin] RBAC: Authorizing `identity:create_grant(project_id=a7010d3512da4f9c9fab7d993427ebf8, user_id=44cef300a7544102a2460ce978c5c9ed, role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:53.651968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18f7f88-65db-4082-8ca1-9097f44048bf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:53.669046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 116/234] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:53 2022] PUT /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/44cef300a7544102a2460ce978c5c9ed/roles/609c7d98c34d4dba86ff458ea0726951 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:57:55.191144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e9629-bb19-4779-8e74-2d48cf444cd1 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.191670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e9629-bb19-4779-8e74-2d48cf444cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.191670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e9629-bb19-4779-8e74-2d48cf444cd1 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.193056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 119/235] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:55.199035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c859a4f9-daf5-442f-bdf4-6ce7618ea887 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.199548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c859a4f9-daf5-442f-bdf4-6ce7618ea887 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.199548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c859a4f9-daf5-442f-bdf4-6ce7618ea887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.223854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c859a4f9-daf5-442f-bdf4-6ce7618ea887 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:55.225071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c859a4f9-daf5-442f-bdf4-6ce7618ea887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:55.289025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:55.310050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 117/236] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:55.352732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c0a3c0-c58b-405b-bd33-b7e1bab7b438 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.353061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c0a3c0-c58b-405b-bd33-b7e1bab7b438 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.353308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c0a3c0-c58b-405b-bd33-b7e1bab7b438 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.354823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 120/237] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:55.360310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afda5cbf-104e-43dc-8aec-c9d7786c97a2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.360850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afda5cbf-104e-43dc-8aec-c9d7786c97a2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.361350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afda5cbf-104e-43dc-8aec-c9d7786c97a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.380105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-afda5cbf-104e-43dc-8aec-c9d7786c97a2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:55.381637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-afda5cbf-104e-43dc-8aec-c9d7786c97a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:55.474034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 118/238] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:55.577287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a4eb0-75cb-4406-9615-96d3ea5e6540 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.577702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a4eb0-75cb-4406-9615-96d3ea5e6540 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.577851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033a4eb0-75cb-4406-9615-96d3ea5e6540 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.579165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 121/239] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:55.588718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.589270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.589736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.616987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:55.618186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:55.706598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 119/240] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:55.751236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033a4eb0-75cb-4406-9615-96d3ea5e6540 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:55.825828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:55.882311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033a4eb0-75cb-4406-9615-96d3ea5e6540 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:55.891730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:55.925240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:55.925878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:55.926157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:55.926432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:55.938665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:55.944721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:55.945654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:57:55.946484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:55.950797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 122/241] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:55.951958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:55.998012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.060248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c23c2d-9e2e-44c4-bbb0-8ecbb525408e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.095129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fbd62a-4766-4d0f-9aa4-0adbf57695e7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.097253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f906bf-d00e-4477-b4c5-deeaeb6086f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.097895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f906bf-d00e-4477-b4c5-deeaeb6086f8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.098175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f906bf-d00e-4477-b4c5-deeaeb6086f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.098450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f906bf-d00e-4477-b4c5-deeaeb6086f8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.099557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f906bf-d00e-4477-b4c5-deeaeb6086f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.101039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f906bf-d00e-4477-b4c5-deeaeb6086f8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:56.103348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 120/242] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:56.104288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215a013b-148e-4ddd-b8e9-4b7aeb532190 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.104574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215a013b-148e-4ddd-b8e9-4b7aeb532190 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.104846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215a013b-148e-4ddd-b8e9-4b7aeb532190 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.106080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 121/243] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:56.110694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215a013b-148e-4ddd-b8e9-4b7aeb532190 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.116730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.131836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.132455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.133105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.133105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.133652 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:56.134066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.135362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:56.148845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 123/244] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:57:56.148845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.229941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215a013b-148e-4ddd-b8e9-4b7aeb532190 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.249003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.257411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1f5a26-cf5e-44b0-a9a3-1cd99eaddcb7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.270020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd672354-bd2b-439d-bf6d-e4587534a09d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.270648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd672354-bd2b-439d-bf6d-e4587534a09d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.270924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd672354-bd2b-439d-bf6d-e4587534a09d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.271194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd672354-bd2b-439d-bf6d-e4587534a09d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.282521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd672354-bd2b-439d-bf6d-e4587534a09d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-dd672354-bd2b-439d-bf6d-e4587534a09d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:56.287526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:56.288493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:57:56.289330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 122/245] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:57:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:56.290517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.291429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.291429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.302036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.302728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.303018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.303295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.304903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.305620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:56.315759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:56.316887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:56.328784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 124/246] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:56 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:56.329442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.386798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.418646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 123/247] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:56.419329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.490909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75b45bc-c2ad-4b1d-87f7-26b2460f3366 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.535655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46d8d7b-749f-4c40-988d-eaa3536d7a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.536468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46d8d7b-749f-4c40-988d-eaa3536d7a62 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.536862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46d8d7b-749f-4c40-988d-eaa3536d7a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.537183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46d8d7b-749f-4c40-988d-eaa3536d7a62 None admin] PATH_INFO: `/v3/users/bbda660a96dd451284ec049af5613076` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.543242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a048b25-19d1-4eb2-8814-1b83c908e98a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.549767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.558029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46d8d7b-749f-4c40-988d-eaa3536d7a62 None admin] RBAC: Authorizing `identity:get_user(user_id=bbda660a96dd451284ec049af5613076)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.559560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46d8d7b-749f-4c40-988d-eaa3536d7a62 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:56.561667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 125/248] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:56 2022] GET /identity/v3/users/bbda660a96dd451284ec049af5613076 => generated 297 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:56.564118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5445b131-e284-4678-862e-d729d09a69ca None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.564118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5445b131-e284-4678-862e-d729d09a69ca None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.564118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5445b131-e284-4678-862e-d729d09a69ca None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.564838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 126/249] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:56.574320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.574564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.574809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.594063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:56.595260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:56.603650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.604559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.604841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.605138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.605812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:56.606115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.607513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:56.620059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 124/250] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:56.620059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.677191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 127/251] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:56.678641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.712700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.750703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d0e81d-7106-4df0-afcc-cd156ba534fb None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.797394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.798178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.798464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.798741 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.805151 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965f8db7-6695-4e29-bce6-b92d9aef8bc7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:56.844494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:56.845093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:56.846196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:56.846692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:56.847127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] PATH_INFO: `/v3/users/44cef300a7544102a2460ce978c5c9ed` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:56.848603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] RBAC: Authorizing `identity:get_user(user_id=44cef300a7544102a2460ce978c5c9ed)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.850880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:56.853097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 128/252] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:56 2022] GET /identity/v3/users/44cef300a7544102a2460ce978c5c9ed => generated 283 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:56.854112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:56.905388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:56.907188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.004926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.013203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d5491a-54e6-44d2-ac44-aa3c1a6d92b5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.057681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.058932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.059775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.060274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.076227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:57.077386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 125/253] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:57.078228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:57.083236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:57.084137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:57.085335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:57.087948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 129/254] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:57.089458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.151117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.190074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae2f5da-e43a-4544-b5de-861b70ae3691 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.209821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e293f-ce5b-4323-af44-9db3017718c4 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.232796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.233447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.233734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.234010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.246921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.247555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.247827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.248093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.259950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:57.265430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:57.266285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:57:57.267072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:57.271804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 130/255] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:57.272637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.291632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.308407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.310094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.380112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.391093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c78a7e3-b281-46f1-988e-dd4dcce1adce None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.422093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 126/256] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:57.422867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.445527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.446167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.446502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.446772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.448414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.450200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.452837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 131/257] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:57.453615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.553334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd46643c-bb73-4794-b57a-0df6eece5899 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.553953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.578063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e7a15f-beff-45b1-9ea7-0a80173ff8f2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.600059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.600680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.601009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.601285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.602019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:57.602327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.605163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.617614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.618286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.618563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.618833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.619509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:57.619808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.620091 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 127/258] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:57.621157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.621270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.636238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 132/259] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:57.637472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.654812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.741095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9f2c9-2938-45ac-af9f-f2a7a294e820 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.745509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3acb8a-e27b-4feb-915a-e857ffff8b78 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.781578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.782283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.782659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.782973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.783570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.783758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] RBAC: Adding query filter params (role.id=bf23174ef65e4c6a817a0049740656f1, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=bbda660a96dd451284ec049af5613076) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:57.783995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.784196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.784479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.784755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.785458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.786013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.787713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.793028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 128/260] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:57.793899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.797071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 133/261] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/role_assignments?user.id=bbda660a96dd451284ec049af5613076&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=bf23174ef65e4c6a817a0049740656f1 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:57.798113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:57.811921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:57.952594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57d7204-2d4d-4e5e-ba0d-f5b23a390351 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:57.994646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:57.994646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:57.994646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:57.994940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] PATH_INFO: `/v3/users/44cef300a7544102a2460ce978c5c9ed` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:57.995884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] RBAC: Authorizing `identity:get_user(user_id=44cef300a7544102a2460ce978c5c9ed)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:57.997468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:57.999860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 134/262] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/users/44cef300a7544102a2460ce978c5c9ed => generated 283 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:58.000513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:58.021664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:58.029161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480d4299-543f-4335-82ec-3076b19557e7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:58.094962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:58.095937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.096413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.096895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] PATH_INFO: `/v3/users/e53c94d0230b46879f98f2f0fe8d34ca` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.098463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] RBAC: Authorizing `identity:get_user(user_id=e53c94d0230b46879f98f2f0fe8d34ca)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:58.100107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:58.103576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 129/263] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/users/e53c94d0230b46879f98f2f0fe8d34ca => generated 297 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:58.105707 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:58.136203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:58.170037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ee9dd3-ff10-4241-ace0-7e9b88482817 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:58.235024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:58.235637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.235916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.236191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.236930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=44cef300a7544102a2460ce978c5c9ed) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:58.237244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:58.238689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:58.262252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 135/264] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:57 2022] GET /identity/v3/role_assignments?user.id=44cef300a7544102a2460ce978c5c9ed&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:58.263232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:58.311279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7e9098-7e71-40e3-92b0-eb45356d84c0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:58.384519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:58.385533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.386032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.386494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.470810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:58.471513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f49667-5aa7-4c71-a25f-19eae4376f42 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:58.513961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:58.515758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:58.534809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:58.536155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.536638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.537249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.538721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:58.541200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:58.543543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 136/265] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:58 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:58.551093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:58.572446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:58.725118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 130/266] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:57:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 619 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:57:58.734455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:58.762776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22debd5-2be5-4725-928f-9820b04f185b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:58.809300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697b041-2d07-438c-85fd-04691ae1f80e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:58.809890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697b041-2d07-438c-85fd-04691ae1f80e None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.810423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697b041-2d07-438c-85fd-04691ae1f80e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.810697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697b041-2d07-438c-85fd-04691ae1f80e None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/e53c94d0230b46879f98f2f0fe8d34ca/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.813441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697b041-2d07-438c-85fd-04691ae1f80e None admin] RBAC: Authorizing `identity:create_grant(project_id=a7010d3512da4f9c9fab7d993427ebf8, user_id=e53c94d0230b46879f98f2f0fe8d34ca, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:58.815411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697b041-2d07-438c-85fd-04691ae1f80e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:58.834163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 137/267] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:57:58 2022] PUT /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/e53c94d0230b46879f98f2f0fe8d34ca/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:57:58.836547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9258664-dfab-406d-9719-5a9d2fda953f None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.839047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9258664-dfab-406d-9719-5a9d2fda953f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.839325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9258664-dfab-406d-9719-5a9d2fda953f None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.840961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 138/268] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:58.842883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:58.853691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1c41a-83b1-4b16-99a4-cdd8437ba3ca None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.853999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1c41a-83b1-4b16-99a4-cdd8437ba3ca None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.854226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1c41a-83b1-4b16-99a4-cdd8437ba3ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.894070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-18a1c41a-83b1-4b16-99a4-cdd8437ba3ca None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:58.895489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-18a1c41a-83b1-4b16-99a4-cdd8437ba3ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:58.913694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f264e8b-25df-4ae3-9cdf-3bfda0c0158e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:58.962983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:58.963799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:58.964085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:58.964356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:58.965849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:58.968708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:58.970624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 131/269] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:57:58 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:58.977814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:59.043464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:59.049523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 139/270] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:57:59.113842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f260fdbe-72ce-4774-9144-68d84126173b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:59.146635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:59.152478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:59.153113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:59.153391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:59.153666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:59.154391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] RBAC: Adding query filter params (role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=44cef300a7544102a2460ce978c5c9ed) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:57:59.154698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:59.156134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcad528-f1df-4d9f-a47e-b8a844423c33 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:59.162482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 132/271] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:57:58 2022] GET /identity/v3/role_assignments?user.id=44cef300a7544102a2460ce978c5c9ed&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:59.289436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486624a6-6141-41f1-9ff1-89b007c3acc9 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:59.289436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486624a6-6141-41f1-9ff1-89b007c3acc9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:59.289436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486624a6-6141-41f1-9ff1-89b007c3acc9 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:59.290927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 140/272] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:57:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:57:59.296937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:59.297318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:59.297660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:59.326070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:57:59.327408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:57:59.331252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486624a6-6141-41f1-9ff1-89b007c3acc9 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:59.451940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486624a6-6141-41f1-9ff1-89b007c3acc9 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:59.467461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 133/273] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:57:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:57:59.489292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:59.489923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:59.490198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:59.490462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:59.491559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:59.493027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:59.494711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 141/274] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:57:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:57:59.500191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:59.608898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa7493a-fcd5-4530-be03-e0491d0795a4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:59.637871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:59.646897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:59.647500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:59.647780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:59.648100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:59.648758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:57:59.650931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:57:59.677769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 134/275] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:57:59 2022] POST /identity/v3/users => generated 293 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:57:59.771706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:57:59.856314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:57:59.896236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb38b8d-38bf-4f5c-b9e6-a56fbc1a463d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:57:59.935561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:57:59.936204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:57:59.936483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:57:59.936750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:57:59.947722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:57:59.952511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:57:59.953432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:57:59.954230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 142/276] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:57:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:57:59.959696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:00.074636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93031c5-b7a0-4106-81a7-4f5244e167fb None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:00.112030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:00.112636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:00.112943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:00.113215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:00.113872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:00.114181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:00.115537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:00.124575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 135/277] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:57:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:00.130669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:00.243076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539a1757-6e73-4533-a5c2-91f1297610a3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:00.284079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:00.296385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9381f28-ecb3-43ab-85dc-48e8331acfed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:00.297154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9381f28-ecb3-43ab-85dc-48e8331acfed None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:00.297367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9381f28-ecb3-43ab-85dc-48e8331acfed None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:00.297641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9381f28-ecb3-43ab-85dc-48e8331acfed None admin] PATH_INFO: `/v3/users/1d21d63996aa4f91ba37c09807a5b82b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:00.312643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9381f28-ecb3-43ab-85dc-48e8331acfed None admin] RBAC: Authorizing `identity:get_user(user_id=1d21d63996aa4f91ba37c09807a5b82b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:00.314183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9381f28-ecb3-43ab-85dc-48e8331acfed None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:00.316221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 143/278] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:00 2022] GET /identity/v3/users/1d21d63996aa4f91ba37c09807a5b82b => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:00.322996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:00.378386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:00.450519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e858d85-9739-4438-96aa-847630db5525 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:00.501544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:00.502158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:00.502439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:00.502703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:00.503423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, user.id=1d21d63996aa4f91ba37c09807a5b82b, scope.system=all) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:00.503728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:00.503821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e32bf7a-d259-4e0e-816a-61da76efe681 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:00.504081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e32bf7a-d259-4e0e-816a-61da76efe681 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:00.504358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e32bf7a-d259-4e0e-816a-61da76efe681 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:00.505188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:00.505703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 144/279] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:00.510554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 136/280] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 29 01:58:00 2022] GET /identity/v3/role_assignments?user.id=1d21d63996aa4f91ba37c09807a5b82b&scope.system=all&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:00.518664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:00.519303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:00.519787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:00.549618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:00.550851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:00.630693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 145/281] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:01.030506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:01.171270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e79642-b5e1-4296-bf64-d8dde0ce0033 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:01.180700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:01.206636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:01.207282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.207562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.207836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.218575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:01.223760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:01.224617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:01.225454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:01.228973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 137/282] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:01.231627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:01.338478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ad21dc-0edb-4d25-bfbc-948e83b616e5 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:01.353022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:01.373553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:01.374164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.374441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.374711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.375888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:01.376188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:01.381025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:01.394435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 146/283] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:01 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:01.395815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:01.522830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e095cef5-9031-4601-844e-6965252ba8f6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:01.562423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8559e485-c8c5-4733-8502-806009c3d409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:01.563067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8559e485-c8c5-4733-8502-806009c3d409 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.563348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8559e485-c8c5-4733-8502-806009c3d409 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.563623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8559e485-c8c5-4733-8502-806009c3d409 None admin] PATH_INFO: `/v3/users/bbda660a96dd451284ec049af5613076` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.567842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8559e485-c8c5-4733-8502-806009c3d409 None admin] RBAC: Authorizing `identity:get_user(user_id=bbda660a96dd451284ec049af5613076)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:01.567842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8559e485-c8c5-4733-8502-806009c3d409 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:01.573020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 138/284] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:01 2022] GET /identity/v3/users/bbda660a96dd451284ec049af5613076 => generated 297 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:01.574797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:01.595808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a393aa-ee18-49ae-b9ec-8a19db705e31 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.596061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a393aa-ee18-49ae-b9ec-8a19db705e31 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.596295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a393aa-ee18-49ae-b9ec-8a19db705e31 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.597927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 139/285] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:01.606693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.606982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.607262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.631239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:01.632703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:01.692432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:01.728583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63164980-d1b3-443b-9e89-7b4cb5443b0b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:01.758571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 140/286] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:01.791165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:01.791808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.792080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.792345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.793501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:01.794912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:01.796523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 147/287] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:01 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:01.802552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:01.932130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae7064a-0df0-4aa0-a6e7-83719b8f4966 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:01.977542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:01.978728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:01.979007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:01.979279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/bbda660a96dd451284ec049af5613076/roles/bf23174ef65e4c6a817a0049740656f1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:01.986123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] RBAC: Authorizing `identity:create_grant(project_id=a7010d3512da4f9c9fab7d993427ebf8, user_id=bbda660a96dd451284ec049af5613076, role_id=bf23174ef65e4c6a817a0049740656f1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:01.987560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:02.013910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 141/288] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:01 2022] PUT /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/users/bbda660a96dd451284ec049af5613076/roles/bf23174ef65e4c6a817a0049740656f1 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:02.226164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:02.295716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:02.367231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b57741a-d345-44f0-9d37-10a04c10b10b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:02.418911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3895924b-1592-487c-9f33-f196f4008c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:02.419571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895924b-1592-487c-9f33-f196f4008c97 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.419850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895924b-1592-487c-9f33-f196f4008c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.420200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895924b-1592-487c-9f33-f196f4008c97 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.437883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3895924b-1592-487c-9f33-f196f4008c97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-3895924b-1592-487c-9f33-f196f4008c97 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:02.443475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:02.444348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:02.445188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 148/289] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:02.454059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:02.501310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:02.549434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac4e0d-12cc-470f-a747-6b326b85724a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.549755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac4e0d-12cc-470f-a747-6b326b85724a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.549987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac4e0d-12cc-470f-a747-6b326b85724a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.551548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 149/290] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:02.558097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99774ef-e824-4098-a3bd-7fa07c19313c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.558381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99774ef-e824-4098-a3bd-7fa07c19313c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.558644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99774ef-e824-4098-a3bd-7fa07c19313c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.598899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c99774ef-e824-4098-a3bd-7fa07c19313c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:02.600086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c99774ef-e824-4098-a3bd-7fa07c19313c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:02.629574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053877c3-8ec9-4765-bc09-0f7419f5fd6a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:02.670620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:02.671572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.672072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.673078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.673971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:02.674716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:02.676565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e8da1-9e26-465b-b470-ae9ef0169fe2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:02.691697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 142/291] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:02.691697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fc486-76ed-4401-8511-b8f251099641 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.691697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fc486-76ed-4401-8511-b8f251099641 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.691697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fc486-76ed-4401-8511-b8f251099641 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.691697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 143/292] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:02.693623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 150/293] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:02.696052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2fc486-76ed-4401-8511-b8f251099641 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:02.701588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.701876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.702156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.730479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:02.731639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:02.796308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:02.831940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:02.855873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2fc486-76ed-4401-8511-b8f251099641 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:02.860956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 151/294] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:02.891380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:02.892012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:02.892289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:02.892606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] PATH_INFO: `/v3/users/e53c94d0230b46879f98f2f0fe8d34ca` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:02.893786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] RBAC: Authorizing `identity:get_user(user_id=e53c94d0230b46879f98f2f0fe8d34ca)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:02.895218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:02.897184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 144/295] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:02 2022] GET /identity/v3/users/e53c94d0230b46879f98f2f0fe8d34ca => generated 297 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:02.906207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.034256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1724f0-538c-4b27-86a8-fc86140350d2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.043378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.087208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.087607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.087886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.088157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.159215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:03.203322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:03.204908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:03.217299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a0dad6-d3c5-4055-b426-4b0a6ebf842b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.251920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4650be49-7533-4b76-96ab-a5a9b13c9711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.252547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4650be49-7533-4b76-96ab-a5a9b13c9711 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.252825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4650be49-7533-4b76-96ab-a5a9b13c9711 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.253125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4650be49-7533-4b76-96ab-a5a9b13c9711 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.254222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4650be49-7533-4b76-96ab-a5a9b13c9711 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:03.255671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4650be49-7533-4b76-96ab-a5a9b13c9711 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:03.257434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 145/296] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:03.263343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c4bc0-859e-42fb-bdbf-b3ab99a96c57 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.263628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c4bc0-859e-42fb-bdbf-b3ab99a96c57 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.263924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c4bc0-859e-42fb-bdbf-b3ab99a96c57 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.265410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 146/297] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:03.266118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c4bc0-859e-42fb-bdbf-b3ab99a96c57 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.291780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:03.317612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 152/298] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:58:03.317612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.380652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c4bc0-859e-42fb-bdbf-b3ab99a96c57 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.385049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:03.423177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47601a0-417f-4f66-810c-6351ef5e9663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.423810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47601a0-417f-4f66-810c-6351ef5e9663 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.424260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47601a0-417f-4f66-810c-6351ef5e9663 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.424534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47601a0-417f-4f66-810c-6351ef5e9663 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.433450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42edadb-0a36-4b25-9116-1338680b9041 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.437766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47601a0-417f-4f66-810c-6351ef5e9663 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f47601a0-417f-4f66-810c-6351ef5e9663 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:03.443692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:03.444557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:03.445370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 147/299] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:03.446775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.446942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.447194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.466355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:03.467545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:03.472522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.473154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.473431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.473697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] PATH_INFO: `/v3/groups` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.474308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:03.475730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:03.496903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 153/300] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Sep 29 01:58:03 2022] POST /identity/v3/groups => generated 229 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:03.496903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.559553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 148/301] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:03.577134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.617252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894827a3-f1e2-41f5-96c8-f78d49fe9a75 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.617900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:03.651581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:03.664608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.665481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.665773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.666053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.667227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:03.668727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:03.672487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 154/302] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:03.678884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.772322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa8b48-489a-4d82-be11-00778ba1b317 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.816654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d865749f-f923-410d-a21b-00609183438d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.817704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865749f-f923-410d-a21b-00609183438d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.818464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865749f-f923-410d-a21b-00609183438d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.819218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d865749f-f923-410d-a21b-00609183438d None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.820056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d865749f-f923-410d-a21b-00609183438d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:03.820592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d865749f-f923-410d-a21b-00609183438d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:03.820944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a01b55-be72-41ef-a455-a345ba5e3200 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:03.822923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d865749f-f923-410d-a21b-00609183438d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:03.832057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 149/303] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:03.841364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d865749f-f923-410d-a21b-00609183438d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.863817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:03.864762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:03.865222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:03.865500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:03.866234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=e53c94d0230b46879f98f2f0fe8d34ca) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:03.866545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:03.868566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:03.876404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 155/304] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/role_assignments?user.id=e53c94d0230b46879f98f2f0fe8d34ca&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:03.895298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:03.962801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:04.015761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:04.020110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21605e27-a611-41ea-bf07-f56375b789ce None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:04.066949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:04.067887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.068322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.068741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.080306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d865749f-f923-410d-a21b-00609183438d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:04.092384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:04.105798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:04.106743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:04.107542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:04.110438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 156/305] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:04.115870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:04.127296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:04.127938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.128217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.128490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] PATH_INFO: `/v3/users/1ff9956b7be54cd5903cd6fc30b09077` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.146296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] RBAC: Authorizing `identity:get_user(user_id=1ff9956b7be54cd5903cd6fc30b09077)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:04.147772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:04.150011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 150/306] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:03 2022] GET /identity/v3/users/1ff9956b7be54cd5903cd6fc30b09077 => generated 293 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:04.157290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:04.257395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45e9545-d3ec-4077-a8f8-1dfd1d2b79cc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:04.283142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df50eaa3-d48e-43c7-89a8-a767087ccbd9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:04.283779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:04.318285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:04.319767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.320544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.321175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.322100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:04.322763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:04.324922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:04.330265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:04.331075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.331455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.331838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.333311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] RBAC: Adding query filter params (role.id=bf23174ef65e4c6a817a0049740656f1, user.id=1ff9956b7be54cd5903cd6fc30b09077, scope.system=all) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:04.333727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:04.336202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:04.338477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 157/307] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:04.338966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:04.343250 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:04.344418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 151/308] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 29 01:58:04 2022] GET /identity/v3/role_assignments?user.id=1ff9956b7be54cd5903cd6fc30b09077&scope.system=all&role.id=bf23174ef65e4c6a817a0049740656f1 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:04.470270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da1abc-22b4-49d0-9502-e2d277ac008f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:04.507914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d78fb1-5393-438c-af13-35781f57b08f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:04.508479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d78fb1-5393-438c-af13-35781f57b08f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.508785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d78fb1-5393-438c-af13-35781f57b08f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.509116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d78fb1-5393-438c-af13-35781f57b08f None admin] PATH_INFO: `/v3/users/1d21d63996aa4f91ba37c09807a5b82b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.510283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d78fb1-5393-438c-af13-35781f57b08f None admin] RBAC: Authorizing `identity:get_user(user_id=1d21d63996aa4f91ba37c09807a5b82b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:04.511749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d78fb1-5393-438c-af13-35781f57b08f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:04.513742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 158/309] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:04 2022] GET /identity/v3/users/1d21d63996aa4f91ba37c09807a5b82b => generated 293 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:04.519337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:04.624487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0209efa3-5abe-466c-b44f-7d96ec6ef0dc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:04.662029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aced6c7-d70b-4531-9e29-1a0617cd885d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:04.662694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aced6c7-d70b-4531-9e29-1a0617cd885d None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.662973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aced6c7-d70b-4531-9e29-1a0617cd885d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.663307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aced6c7-d70b-4531-9e29-1a0617cd885d None admin] PATH_INFO: `/v3/system/users/1d21d63996aa4f91ba37c09807a5b82b/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.664932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aced6c7-d70b-4531-9e29-1a0617cd885d None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1d21d63996aa4f91ba37c09807a5b82b, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:04.666937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aced6c7-d70b-4531-9e29-1a0617cd885d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:04.685308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 152/310] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Sep 29 01:58:04 2022] PUT /identity/v3/system/users/1d21d63996aa4f91ba37c09807a5b82b/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:04.772713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f8a77-a86f-4c8b-99b2-b8661b2f7a99 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.772713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f8a77-a86f-4c8b-99b2-b8661b2f7a99 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.772713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f8a77-a86f-4c8b-99b2-b8661b2f7a99 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.773309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 159/311] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:04.779221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:04.779901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:04.780377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:04.800251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:04.801863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:04.865810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:04.880013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 153/312] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:05.151638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059f8a77-a86f-4c8b-99b2-b8661b2f7a99 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:05.203433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:05.259982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059f8a77-a86f-4c8b-99b2-b8661b2f7a99 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:05.297439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:05.298069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:05.298347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:05.298616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:05.308956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:05.313978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:05.315104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:05.315885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 160/313] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:05.321116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:05.430632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021ac69f-ae6c-495a-aec0-c6d0137c9e04 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:05.468705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:05.469346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:05.469624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:05.469894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:05.470557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:05.470858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:05.472224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:05.480815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 154/314] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:05.487064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:05.600303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69c230-ca2f-4410-9564-6661e1bdeb6e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:05.610535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:05.640128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:05.641424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:05.641701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:05.641969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] PATH_INFO: `/v3/users/bbda660a96dd451284ec049af5613076` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:05.643779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] RBAC: Authorizing `identity:get_user(user_id=bbda660a96dd451284ec049af5613076)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:05.645279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:05.647193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 161/315] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:05 2022] GET /identity/v3/users/bbda660a96dd451284ec049af5613076 => generated 297 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:05.653069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:05.722006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:05.757890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e0865-ba76-459c-9829-0949070ccf79 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:05.792830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ff6a6a-288f-46a4-a9a6-a9228b18faa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:05.793781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff6a6a-288f-46a4-a9a6-a9228b18faa4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:05.794257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff6a6a-288f-46a4-a9a6-a9228b18faa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:05.794721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff6a6a-288f-46a4-a9a6-a9228b18faa4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:05.863214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ff6a6a-288f-46a4-a9a6-a9228b18faa4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:05.865018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ff6a6a-288f-46a4-a9a6-a9228b18faa4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:05.918814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:05.966321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 155/316] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:05.974615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:06.075573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ee8627-2b02-4177-b854-40523294741c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:06.100189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75cae40-2988-4727-b56f-ae95d1ad2242 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.100514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75cae40-2988-4727-b56f-ae95d1ad2242 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.100745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75cae40-2988-4727-b56f-ae95d1ad2242 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.102376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 156/317] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:06.110001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.110404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.110778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.112703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:06.113515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.113928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.114412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.115874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:06.117496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:06.119314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 162/318] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:05 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:06.125499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:06.132453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:06.134473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:06.171789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:06.217374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 157/319] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:06.230107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f97d76-4164-48aa-b54a-ff36308a6e57 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:06.264187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:06.265137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.265632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.266094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.267006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] RBAC: Adding query filter params (role.id=bf23174ef65e4c6a817a0049740656f1, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8, user.id=bbda660a96dd451284ec049af5613076) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:06.267503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:06.269147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae4304e-4df9-4feb-9d76-be0385286d87 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:06.275376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 163/320] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:06 2022] GET /identity/v3/role_assignments?user.id=bbda660a96dd451284ec049af5613076&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=bf23174ef65e4c6a817a0049740656f1 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:06.488351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:06.588330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:06.605999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34003b3-ab8a-44d0-bfd7-293aef58b1a0 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:06.607093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015510cb-f5d1-40f4-b1fe-d50f6d4cda50 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.607535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015510cb-f5d1-40f4-b1fe-d50f6d4cda50 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.607655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015510cb-f5d1-40f4-b1fe-d50f6d4cda50 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.609186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 164/321] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:06.614125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.614383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.614661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.632279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:06.633505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:06.641340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:06.641979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.642263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.642539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.643645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:06.645185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:06.647033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 158/322] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:06.652567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:06.706742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 165/323] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:06.754905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cbb86b-ba7e-4769-8d6a-6961b1e78383 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:06.787912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3213382-081a-44aa-80d6-429e8df26b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:06.788516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3213382-081a-44aa-80d6-429e8df26b54 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.788796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3213382-081a-44aa-80d6-429e8df26b54 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.789095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3213382-081a-44aa-80d6-429e8df26b54 None admin] PATH_INFO: `/v3/groups` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.789723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3213382-081a-44aa-80d6-429e8df26b54 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:06.791696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3213382-081a-44aa-80d6-429e8df26b54 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:06.808511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 159/324] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Sep 29 01:58:06 2022] POST /identity/v3/groups => generated 226 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:58:06.982785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:06.990523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3b55df-2439-4441-a186-7e65af451b98 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.990523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3b55df-2439-4441-a186-7e65af451b98 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.990523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3b55df-2439-4441-a186-7e65af451b98 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:06.991035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 160/325] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:06.996655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:06.996973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:06.997342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.015365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:07.016473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:07.020181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:07.024329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:07.094255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 161/326] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:07.102057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a3f22e-43d0-4ea0-92a3-d87e821d3eb6 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.145832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.146855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.147347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.147858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.159623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:07.164906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:07.165861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:07.166654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:07.168681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 166/327] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:07.175336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:07.289337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4fc6b-b933-464d-8179-573fe937d474 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.326143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.327071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.327527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.327970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.328806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] RBAC: Adding query filter params (name=reader) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:07.329305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:07.330860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:07.341511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 162/328] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:07.346904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:07.375496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:07.379942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:07.469145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:07.475347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c06d0d-9bb9-4100-a062-527e6274e555 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.488005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5eda6-1fd9-457c-b659-180d8f7262ea None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.521864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.522612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.522892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.523165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] PATH_INFO: `/v3/users/1ff9956b7be54cd5903cd6fc30b09077` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.524312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] RBAC: Authorizing `identity:get_user(user_id=1ff9956b7be54cd5903cd6fc30b09077)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:07.525829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:07.527772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 167/329] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/users/1ff9956b7be54cd5903cd6fc30b09077 => generated 293 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:07.531805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.532555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.532834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.533128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.534405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:07.552329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:07.557913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:07.558771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:07.559557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:07.560146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 163/330] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:07.572124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:07.644757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:07.648563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329d542-e380-45c0-82b1-ebeafe379ddd None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.689486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7c8ca8-fd94-4ecb-93cb-4fd347bf1a57 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.691040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.691857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.692277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.692712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] PATH_INFO: `/v3/system/users/1ff9956b7be54cd5903cd6fc30b09077/roles/bf23174ef65e4c6a817a0049740656f1` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.694786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1ff9956b7be54cd5903cd6fc30b09077, role_id=bf23174ef65e4c6a817a0049740656f1)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:07.696341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:07.708834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 168/331] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Sep 29 01:58:07 2022] PUT /identity/v3/system/users/1ff9956b7be54cd5903cd6fc30b09077/roles/bf23174ef65e4c6a817a0049740656f1 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:07.735001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.735692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.736005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.736284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.737101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:07.737409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:07.739022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:07.749803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 164/332] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:07.755340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:07.791253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:07.875630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768540ce-3010-4b6f-9bc2-a036a08ddc43 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:07.914044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057d220a-660d-4e75-a5b1-ed3b564c8b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:07.914955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057d220a-660d-4e75-a5b1-ed3b564c8b88 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.915444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057d220a-660d-4e75-a5b1-ed3b564c8b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.915856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057d220a-660d-4e75-a5b1-ed3b564c8b88 None admin] PATH_INFO: `/v3/users/1d21d63996aa4f91ba37c09807a5b82b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.917328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057d220a-660d-4e75-a5b1-ed3b564c8b88 None admin] RBAC: Authorizing `identity:get_user(user_id=1d21d63996aa4f91ba37c09807a5b82b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:07.919139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057d220a-660d-4e75-a5b1-ed3b564c8b88 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:07.921404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 169/333] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/users/1d21d63996aa4f91ba37c09807a5b82b => generated 293 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:07.927226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:08.040920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395dff7-da33-4864-84a7-b6f3603be747 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:08.069283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:08.082984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:08.083597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:08.083923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:08.084199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:08.084949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] RBAC: Adding query filter params (role.id=ce237b90aca34abead4f05dfc46a1ec4, user.id=1d21d63996aa4f91ba37c09807a5b82b, scope.system=all) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:08.085260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:08.086682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f7f46e-42d9-4f5b-9f23-2f0078f15233 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:08.099796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 165/334] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/role_assignments?user.id=1d21d63996aa4f91ba37c09807a5b82b&scope.system=all&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 510 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:09.381621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6415cac-c05a-49d2-a739-33be9d21b38e None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:09.382618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6415cac-c05a-49d2-a739-33be9d21b38e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:09.383126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6415cac-c05a-49d2-a739-33be9d21b38e None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:09.385030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 170/335] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:09.390660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8144a-72f8-4384-9d56-9e654f8c5640 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:09.391485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8144a-72f8-4384-9d56-9e654f8c5640 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:09.391983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8144a-72f8-4384-9d56-9e654f8c5640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:09.412184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-18a8144a-72f8-4384-9d56-9e654f8c5640 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:09.413764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-18a8144a-72f8-4384-9d56-9e654f8c5640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:09.504265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 166/336] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:09.586615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6932d6-39f9-41c3-8715-ea48935c3541 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:09.587419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6932d6-39f9-41c3-8715-ea48935c3541 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:09.587701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6932d6-39f9-41c3-8715-ea48935c3541 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:09.589082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 171/337] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:09.595574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:09.595870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:09.596153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:09.625205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:09.626415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:09.715999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 167/338] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:09.915254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6932d6-39f9-41c3-8715-ea48935c3541 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.023552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:10.032410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6932d6-39f9-41c3-8715-ea48935c3541 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:10.069694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:10.070329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:10.070610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:10.070946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:10.087311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:10.092355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:10.093249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:10.094036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 172/339] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:10.099800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.151564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:10.224404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2c8cbb-5d84-4e81-a3f3-9a3e8d91968b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:10.269804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:10.270869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:10.271378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:10.271881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:10.272658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:10.272990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:10.274477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:10.283692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 168/340] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:10.289611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.393749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.463729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b603852-a0be-45eb-8aab-15faa57bf6ad None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:10.537170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:10.537710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:10.537994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:10.538272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:10.559901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aa3b2e-410a-487b-aee3-d74e58bb29f8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:10.574331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:10.606128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:10.619487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:10.620132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:10.620406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:10.620674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:10.636638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:10.646210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:10.647162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:10.647980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 169/341] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:10.655179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.674783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:10.676298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:10.807715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083c13d6-6445-4240-998c-d1a654b0c65f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:10.816567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 173/342] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:10.823015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.860278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:10.860915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:10.861197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:10.861467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:10.862135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:10.862436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:10.863816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:10.880082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 170/343] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:10.885646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:10.905659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:10.931612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:10.981146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3362eab-7553-441a-99ac-8d3e9d6bc60b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:11.035153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:11.035811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.036092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.036366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.038320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1562ef74-0def-4235-89f1-2e80352021aa None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:11.041667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:11.044923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:11.044923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 174/344] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:10 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:11.059818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:11.093368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0984c1a-5515-404b-9dfd-774fa491bf2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:11.094418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0984c1a-5515-404b-9dfd-774fa491bf2c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.094900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0984c1a-5515-404b-9dfd-774fa491bf2c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.095367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0984c1a-5515-404b-9dfd-774fa491bf2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.210414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0984c1a-5515-404b-9dfd-774fa491bf2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:11.212758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0984c1a-5515-404b-9dfd-774fa491bf2c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:11.246189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:11.247115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9373be6b-5a00-460b-ab17-c12a65dda856 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:11.306379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:11.331461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:11.332030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.332305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.332575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.341128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:11.342537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 171/345] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:58:11.342647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:11.344083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affd9a4d-1120-4a59-a48e-cb6f50e9aea5 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.344643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affd9a4d-1120-4a59-a48e-cb6f50e9aea5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.345210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affd9a4d-1120-4a59-a48e-cb6f50e9aea5 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.346648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 172/346] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:11.347837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d78100-b818-4bcd-85ac-98b25e013d3f None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.348314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d78100-b818-4bcd-85ac-98b25e013d3f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.348959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d78100-b818-4bcd-85ac-98b25e013d3f None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.349658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 175/347] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:11 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:11.350486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:11.350957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 173/348] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:11.358212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4159894a-194a-478b-a814-2352af910b73 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.358892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4159894a-194a-478b-a814-2352af910b73 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.359363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4159894a-194a-478b-a814-2352af910b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.384258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4159894a-194a-478b-a814-2352af910b73 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:11.385490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4159894a-194a-478b-a814-2352af910b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:11.474411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 174/349] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:11.476841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4159894a-194a-478b-a814-2352af910b73 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:11.527292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f643a784-98f4-4290-bb0f-ed0e08010fed None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:11.582815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:11.602458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65239322-ae8c-4710-840b-0379e419140a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:11.603687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65239322-ae8c-4710-840b-0379e419140a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.603963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65239322-ae8c-4710-840b-0379e419140a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.604233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65239322-ae8c-4710-840b-0379e419140a None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.606261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65239322-ae8c-4710-840b-0379e419140a None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:11.607701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65239322-ae8c-4710-840b-0379e419140a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:11.611003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 176/350] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:11 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:11.615840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586c3159-f639-41c1-a0f2-9c5bbc0bc885 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.616128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586c3159-f639-41c1-a0f2-9c5bbc0bc885 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.616396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586c3159-f639-41c1-a0f2-9c5bbc0bc885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.656893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-586c3159-f639-41c1-a0f2-9c5bbc0bc885 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:11.658099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-586c3159-f639-41c1-a0f2-9c5bbc0bc885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:11.659434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4159894a-194a-478b-a814-2352af910b73 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:11.707491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:11.708197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.708616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.708955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.709863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=881f2ae0820241ec9d3216765931ba06) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:11.710241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:11.712514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:11.718294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 175/351] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:11 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 260 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:11.719217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:11.729301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:11.747502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 177/352] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:11.748918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef17a95-c5cf-4e8c-bbc3-1c2cce93c731 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.749212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef17a95-c5cf-4e8c-bbc3-1c2cce93c731 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.749494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef17a95-c5cf-4e8c-bbc3-1c2cce93c731 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.753912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 178/353] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:11.757529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.757815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.758087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.797064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:11.798583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:11.911966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f86bf39-7b3c-487a-a533-fdb885717256 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:11.918308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 179/354] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:11.919863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:11.938218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:11.961978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:11.963019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:11.963443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:11.963844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:11.966343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:11.967974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:11.972822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 176/355] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:11 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:11.980174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.089205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c63f7c-2983-4424-bf58-be0ca25a2827 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.126534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.127472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.127941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.128354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.129272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=881f2ae0820241ec9d3216765931ba06) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:12.129725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:12.131407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:12.138144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 177/356] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:11 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 260 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:12.157002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513f4215-b10b-4194-a67c-9c3369dad5eb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.181899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:12.206756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.207657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.208105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.208520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.226674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:12.231005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:12.243451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:12.244757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:12.246418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 180/357] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:12.253658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.311789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:12.387278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d8129-c4ce-4340-91ed-bb3529ebe37a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.424798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.425833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.426309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.426798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.434369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03e0f19-f79e-48af-8969-797f87fce23b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.441556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:12.447388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:12.448315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:12.450570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:12.452644 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 178/358] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:12.453655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.490514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.491135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.491449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.491924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.492364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:12.492666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:12.494755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6249da1-233c-49ea-93c5-4aa94506bff2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:12.504928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 181/359] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:12.506493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e725867b-11d4-404e-a020-2752ba002de9 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.506787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e725867b-11d4-404e-a020-2752ba002de9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.507066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e725867b-11d4-404e-a020-2752ba002de9 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.508583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 182/360] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:12.509751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e725867b-11d4-404e-a020-2752ba002de9 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.557443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:12.577129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730d787f-4a69-4917-99aa-43f805e31a8f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.597917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:12.622282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.623236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.623706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.624146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.638321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:12.644110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:12.645092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:12.645884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:12.646598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e725867b-11d4-404e-a020-2752ba002de9 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.648712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 179/361] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:12.650492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.686750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.687125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.687413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.687689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.688381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:12.688651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:12.690309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e593d-52d9-4b81-8b4d-bc8b38c8e32a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:12.702502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 183/362] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:12.703811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.704103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.704384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.725044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:12.726245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:12.764540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd521548-99c5-4411-be35-5fc8f8c7397d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.798604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 184/363] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:12.799770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:12.802764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.803608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.804087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.804553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.833281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:12.855451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:12.884519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:12.886381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:12.911168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a340b86-1118-43ca-a787-5864e0d0fb19 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:12.948266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:12.949030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:12.949161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:12.949378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:12.950055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:12.950409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:12.951746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:12.965617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 185/364] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:12.966630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.015343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 180/365] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:13.016910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.129260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.150564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b8d01-f8aa-4464-b0e2-b7d2fe77fb11 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.151469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.198583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77036d2-206f-43ac-aea7-8314818abe15 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.208494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.209173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.209457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.209880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] PATH_INFO: `/v3/users/1ff9956b7be54cd5903cd6fc30b09077` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.211123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] RBAC: Authorizing `identity:get_user(user_id=1ff9956b7be54cd5903cd6fc30b09077)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.212731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.214823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 186/366] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:12 2022] GET /identity/v3/users/1ff9956b7be54cd5903cd6fc30b09077 => generated 293 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:13.215604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.242666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.243888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.244118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.244388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.326294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.328211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.333525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62610803-cd3c-4aab-a1af-6f746a841764 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.370362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.371000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.371279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.371577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.372795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.374419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.376172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 187/367] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:13.376961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.408894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.412231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.444228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 181/368] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:13.445347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.493682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531504e-fedd-4aa6-9dec-2c2bdeac6d4a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.532374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.533472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.533957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.534444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.548164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:13.555428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:13.557257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:13.558544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:13.560826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 188/369] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:13.562408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.574054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7645046-ee62-4bd9-a866-a89efc6d4a20 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.626279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.627567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.627993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.628408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.629855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] RBAC: Adding query filter params (role.id=bf23174ef65e4c6a817a0049740656f1, user.id=1ff9956b7be54cd5903cd6fc30b09077, scope.system=all) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:13.630329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.632499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.651785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 182/370] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/role_assignments?user.id=1ff9956b7be54cd5903cd6fc30b09077&scope.system=all&role.id=bf23174ef65e4c6a817a0049740656f1 => generated 510 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:13.652929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.675609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.694533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb84111-7713-4b09-a83b-eadb1f2303a7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.699355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.741796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.742604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.742983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.743392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.744785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.746373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.748084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 189/371] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:13.749459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.792032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6370133d-317f-4388-8cfa-f64b9eaec49e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.831475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.832295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.832671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.833155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.834395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.835981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.838119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 183/372] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:13.838997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.882210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40201e0-a207-472c-aaaf-ecfc26f5f268 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.927058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:13.927972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:13.928749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:13.929187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:13.930335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:13.931333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:13.933234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:13.945575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 190/373] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:13.947215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:13.955048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:13.969231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556eb4dd-e54e-4214-b455-30c885fcca17 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:13.991832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:14.026899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:14.027508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.027780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.028054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.028784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:14.029115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:14.030539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:14.038822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 184/374] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 260 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:14.039630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:14.115969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b2c8f1-408c-43b3-a5ab-4cbbf48b6e1b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:14.169565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:14.170211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.170491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.170757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.173622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:14.176738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:14.183704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 191/375] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:13 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:14.187278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:14.242160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce7712d-35d5-4495-85f5-892657cc3da5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:14.283267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:14.283802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.284076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.284341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.328283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:14.336098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:14.355568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d865c3b-0789-41f3-9201-d7e324d30166 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:14.379394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:14.380854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:14.414933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:14.416084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.416361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.416626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.417386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:14.417976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:14.419706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:14.426627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 192/376] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:14 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 260 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:14.542497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 185/377] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:14.546098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:14.696616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf7c18-6341-47a9-b452-0456052ce194 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:14.738852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c7134b-9eb8-4f2f-9419-28b78c3b09fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:14.739485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c7134b-9eb8-4f2f-9419-28b78c3b09fb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.739759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c7134b-9eb8-4f2f-9419-28b78c3b09fb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.740026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c7134b-9eb8-4f2f-9419-28b78c3b09fb None admin] PATH_INFO: `/v3/projects/304c048daacb48678f2d64fc933b378e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.741179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c7134b-9eb8-4f2f-9419-28b78c3b09fb None admin] RBAC: Authorizing `identity:get_project(project_id=304c048daacb48678f2d64fc933b378e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:14.742633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c7134b-9eb8-4f2f-9419-28b78c3b09fb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:14.745051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 193/378] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:14 2022] GET /identity/v3/projects/304c048daacb48678f2d64fc933b378e => generated 344 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:14.750417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:14.907178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:14.940848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3e7340-4bf9-4947-ab43-aeb0368ee70e None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.941210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3e7340-4bf9-4947-ab43-aeb0368ee70e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.941465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3e7340-4bf9-4947-ab43-aeb0368ee70e None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.942751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 194/379] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:14.947968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e5553d-ba74-4b48-8466-039979892bbe None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.948243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e5553d-ba74-4b48-8466-039979892bbe None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.948515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e5553d-ba74-4b48-8466-039979892bbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.972170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02b5e34-21fa-4691-90d8-4d3f953543f3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:14.976931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e0e5553d-ba74-4b48-8466-039979892bbe None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:14.978266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e0e5553d-ba74-4b48-8466-039979892bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:15.005975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:15.011625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:15.012273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.012552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.012823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] PATH_INFO: `/v3/groups/39e94a1433754d898647d01c6e0c1376` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.019986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] RBAC: Authorizing `identity:get_group(group_id=39e94a1433754d898647d01c6e0c1376)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:15.021991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:15.026266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 186/380] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:14 2022] GET /identity/v3/groups/39e94a1433754d898647d01c6e0c1376 => generated 229 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:15.032446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.091649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 195/381] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:15.102198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d34f643-4074-4061-bb27-09ac51c3716e None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.103345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d34f643-4074-4061-bb27-09ac51c3716e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.103834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d34f643-4074-4061-bb27-09ac51c3716e None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.105380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 196/382] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:15.111551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.112100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.112562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.148184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:15.149880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:15.195675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b323c0a2-400a-40f5-ac6c-a99546fbdeb9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:15.245891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:15.260784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:15.261760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.262651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.263723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.266042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] RBAC: Adding query filter params (group.id=39e94a1433754d898647d01c6e0c1376, role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=304c048daacb48678f2d64fc933b378e) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:15.266575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:15.268275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:15.268756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 197/383] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:15.282904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 187/384] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/role_assignments?group.id=39e94a1433754d898647d01c6e0c1376&scope.project.id=304c048daacb48678f2d64fc933b378e&role.id=619e79e7bb94488780bd4da040d61c34 => generated 260 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:15.388992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.485511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:15.515743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c8694-291d-48ee-813e-6b4d2acaabab None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:15.554525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:15.555517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.556556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.557089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.573036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:15.577920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:15.578794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:15.579721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 198/385] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:15.585720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.656173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.768920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e45ec90-ab4f-4477-bb4c-72657ceae53b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:15.812081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32162313-6b17-4dae-a81d-b6816b295fa0 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:15.830725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:15.831676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.832155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.832633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.833616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:15.834110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:15.835800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:15.850664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 188/386] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:15.854461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:15.855330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:15.855813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:15.857135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.860123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:15.883066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:15.886051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:15.892633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:15.893762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:15.894763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:15.897789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 199/387] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:15.902030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.967543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:16.018839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82387265-b5f0-408f-9993-151d4bc920ca None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.054350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.054886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.055162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.055429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.056176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:16.056501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.059571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.068603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 200/388] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:16.074691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:16.088063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fb74e-92b5-4fae-a769-95ed1db4309d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.144193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:16.170764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.171750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.172252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.172746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.177651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.179353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.179944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6b39e-d3a7-4f1c-98f6-c302ace45ffe None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.186295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 189/389] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:16.187422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:16.216436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.217161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.217439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.217707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.218917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.220353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.232900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 201/390] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:16 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:16.232900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:16.330614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da08ef7d-1338-444f-8bd1-ef89e87093f2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.336359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d3dee7-8f4a-43a5-ae6d-5e980bfa20ac None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.363490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:16.364756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:16.375937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.376590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.377659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.377940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.379766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.381233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.383017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 202/391] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:16 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:16.393647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.394335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.394614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.394886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.395996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.397680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.400239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 190/392] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:16 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:16.402369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:16.413863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:16.587855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4d610c-59ef-43a2-a114-7fff9820db26 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.604248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b2e9db-12fd-4329-900a-35859d3e4e6b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.633514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3feb3594-434d-4732-a42f-7a6a76d418eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.634243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feb3594-434d-4732-a42f-7a6a76d418eb None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.634525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feb3594-434d-4732-a42f-7a6a76d418eb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.634804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feb3594-434d-4732-a42f-7a6a76d418eb None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06/groups/cabedb4b689143318780aebce32925c9/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.637001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feb3594-434d-4732-a42f-7a6a76d418eb None admin] RBAC: Authorizing `identity:create_grant(project_id=881f2ae0820241ec9d3216765931ba06, group_id=cabedb4b689143318780aebce32925c9, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.638716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3feb3594-434d-4732-a42f-7a6a76d418eb None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.658002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 191/393] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 29 01:58:16 2022] PUT /identity/v3/projects/881f2ae0820241ec9d3216765931ba06/groups/cabedb4b689143318780aebce32925c9/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:16.674733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf7dbe1-9622-48f1-8bb9-5fbc8b6185f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.675450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf7dbe1-9622-48f1-8bb9-5fbc8b6185f3 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.675732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf7dbe1-9622-48f1-8bb9-5fbc8b6185f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.676001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf7dbe1-9622-48f1-8bb9-5fbc8b6185f3 None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06/groups/cabedb4b689143318780aebce32925c9/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.678307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf7dbe1-9622-48f1-8bb9-5fbc8b6185f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=881f2ae0820241ec9d3216765931ba06, group_id=cabedb4b689143318780aebce32925c9, role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.679826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf7dbe1-9622-48f1-8bb9-5fbc8b6185f3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.699610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 203/394] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 29 01:58:16 2022] PUT /identity/v3/projects/881f2ae0820241ec9d3216765931ba06/groups/cabedb4b689143318780aebce32925c9/roles/609c7d98c34d4dba86ff458ea0726951 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:17.123380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63e177-3f36-49f3-9510-766f4bf61ee1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:17.123883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63e177-3f36-49f3-9510-766f4bf61ee1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:17.123980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63e177-3f36-49f3-9510-766f4bf61ee1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:17.125545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 192/395] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:17.134304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:17.134760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:17.134836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:17.158357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:17.159548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:17.181653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09cb98c-45ca-4b21-b5d6-b71aaa6b4eba None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:17.181981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09cb98c-45ca-4b21-b5d6-b71aaa6b4eba None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:17.182206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09cb98c-45ca-4b21-b5d6-b71aaa6b4eba None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:17.183554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 193/396] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:17.189209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:17.189494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:17.189845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:17.223885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:17.225095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:17.247087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:17.268543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 204/397] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:17.322640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:17.355636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 194/398] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:17.769550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:17.829040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:17.890315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b77d5a-aefb-4266-8ea0-246868de039f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:17.959454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:17.960180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:17.960455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:17.960727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:17.973435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:17.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:17.980047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:17.980834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 205/399] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:17.986341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.038400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:18.096208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea9677-9693-4208-891a-e3639dc17301 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.109298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ea46cf-c5e3-46f3-a861-ad58e2cdbd86 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.131699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.132526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.133035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.133475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.134837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:18.135744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.137463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d03934-369d-4405-a3ad-9b5c821c0902 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.146165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 206/400] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:18.147523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ed49b-d61b-463b-bc6d-aa07ea5ca2cc None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.147975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ed49b-d61b-463b-bc6d-aa07ea5ca2cc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.148086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bd13c8-bf3c-40d8-afc0-fa149352daa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.148785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bd13c8-bf3c-40d8-afc0-fa149352daa1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.153107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bd13c8-bf3c-40d8-afc0-fa149352daa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.153107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bd13c8-bf3c-40d8-afc0-fa149352daa1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.153537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ed49b-d61b-463b-bc6d-aa07ea5ca2cc None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.155038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 207/401] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:18.156609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ed49b-d61b-463b-bc6d-aa07ea5ca2cc None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.164609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bd13c8-bf3c-40d8-afc0-fa149352daa1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-21bd13c8-bf3c-40d8-afc0-fa149352daa1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:18.171348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:18.172234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:18.173079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 195/402] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:18.174529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.174808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.175149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.198098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:18.199515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:18.239221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:18.278236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ed49b-d61b-463b-bc6d-aa07ea5ca2cc None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.284210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 196/403] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:18.293435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:18.294700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.319050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.319689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.320022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.320305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.321521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.322952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.324928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 208/404] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:18.330385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.440782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bc6365-b011-47ab-9e94-3f8db574cc2d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.476404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.477189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.477543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.477869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.478986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.480920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.483036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 209/405] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:18.488655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.544588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7004536-60db-4e06-99f2-a868058781f3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.545320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:18.595371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.595993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.596277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.596553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.600145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] RBAC: Adding query filter params (name=member) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:18.600459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.601859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.607285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4913594-7b05-43d8-9bc0-219bb5701fcf None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.612268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 197/406] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:18.613469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.644070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.644781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.645097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.645424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/groups/cabedb4b689143318780aebce32925c9/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.647605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] RBAC: Authorizing `identity:create_grant(project_id=a7010d3512da4f9c9fab7d993427ebf8, group_id=cabedb4b689143318780aebce32925c9, role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.649167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.652588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:18.676176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 210/407] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 29 01:58:18 2022] PUT /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/groups/cabedb4b689143318780aebce32925c9/roles/609c7d98c34d4dba86ff458ea0726951 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:18.677211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.786747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d25bf9-799c-4b7c-9d70-af5466007d98 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.859975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.860626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.860914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.861191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.878876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b23c06-23ad-4c61-a13c-d36c23f621a2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.881040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:18.885201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:18.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:18.888305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:18.889136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:18.893593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 198/408] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:18.896704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.937741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.938386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.939449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.939737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.941371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.943825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.945696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 211/409] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:18.952450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:19.026436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:19.038595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfd5b02-726a-40c9-af33-ff3713e4f0d1 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:19.070596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959e2d7-836c-4c14-adc7-cac831428036 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:19.083363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:19.084218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.084802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.085237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.086536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:19.087633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:19.089738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:19.101363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 199/410] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:19.108916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:19.109897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:19.110795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.111679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.113090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.115278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:19.118695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:19.128382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 212/411] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:19.133887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:19.225267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:19.299239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3ffaa1-4ce3-4c29-9b84-498adce850f4 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:19.319386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d56de6-26b3-4222-a8c1-cbcb453809c2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:19.344180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37eeb36-7ac9-41a4-b1e3-ba927f49ea23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:19.345252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37eeb36-7ac9-41a4-b1e3-ba927f49ea23 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.345725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37eeb36-7ac9-41a4-b1e3-ba927f49ea23 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.346189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37eeb36-7ac9-41a4-b1e3-ba927f49ea23 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/groups/cabedb4b689143318780aebce32925c9/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.348813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37eeb36-7ac9-41a4-b1e3-ba927f49ea23 None admin] RBAC: Authorizing `identity:create_grant(project_id=a7010d3512da4f9c9fab7d993427ebf8, group_id=cabedb4b689143318780aebce32925c9, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:19.350660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37eeb36-7ac9-41a4-b1e3-ba927f49ea23 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:19.367778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 213/412] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 29 01:58:19 2022] PUT /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8/groups/cabedb4b689143318780aebce32925c9/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:19.386867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aea528d-912d-4d99-8bcb-2a7943939042 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.387231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aea528d-912d-4d99-8bcb-2a7943939042 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.387430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aea528d-912d-4d99-8bcb-2a7943939042 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.388991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 214/413] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:19.395167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4c63b-297c-4faf-ab34-39501a15b539 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.395449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4c63b-297c-4faf-ab34-39501a15b539 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.395722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4c63b-297c-4faf-ab34-39501a15b539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.399330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:19.401116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.401646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.402142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] PATH_INFO: `/v3/groups/39e94a1433754d898647d01c6e0c1376` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.403664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] RBAC: Authorizing `identity:get_group(group_id=39e94a1433754d898647d01c6e0c1376)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:19.405487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:19.407790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 200/414] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:19 2022] GET /identity/v3/groups/39e94a1433754d898647d01c6e0c1376 => generated 229 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:19.424557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:19.429733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6cd4c63b-297c-4faf-ab34-39501a15b539 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:19.431003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6cd4c63b-297c-4faf-ab34-39501a15b539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:19.478083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:19.584279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 215/415] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:19.611497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1acc5-2a59-4fc6-ae3a-4837d600fee1 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:19.649405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:19.650577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.650857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.651129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] PATH_INFO: `/v3/projects/304c048daacb48678f2d64fc933b378e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.653227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3599bc95-c012-4566-b55c-cbf9ba08ed83 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.653502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3599bc95-c012-4566-b55c-cbf9ba08ed83 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.653769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3599bc95-c012-4566-b55c-cbf9ba08ed83 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.654575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] RBAC: Authorizing `identity:get_project(project_id=304c048daacb48678f2d64fc933b378e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:19.655432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 216/416] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:19.656016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:19.657724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 201/417] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:19 2022] GET /identity/v3/projects/304c048daacb48678f2d64fc933b378e => generated 344 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:19.663998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.664275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.664551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.665221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:19.687207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:19.688687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:19.739175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:19.828712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 217/418] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:19.846041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93ac5f-7be7-49ec-ab2a-63965d2120b2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:19.849185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:19.884042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:19.897589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:19.898260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:19.898534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:19.898986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] PATH_INFO: `/v3/projects/304c048daacb48678f2d64fc933b378e/groups/39e94a1433754d898647d01c6e0c1376/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:19.907498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] RBAC: Authorizing `identity:create_grant(project_id=304c048daacb48678f2d64fc933b378e, group_id=39e94a1433754d898647d01c6e0c1376, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:19.912208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:19.933466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 202/419] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Sep 29 01:58:19 2022] PUT /identity/v3/projects/304c048daacb48678f2d64fc933b378e/groups/39e94a1433754d898647d01c6e0c1376/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:58:20.034015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c5df26-0687-4fa3-84f2-702e4c57d8eb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.074693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.075355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.075631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.075901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.088273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:20.093401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:20.094356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:20.095156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 218/420] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:20.100960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.181251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.207650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:20.213277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4878a5-3f9d-4b3d-98d5-94f5004b6495 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.255477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.256339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.256622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.257032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.257706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:20.258108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:20.259539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:20.269619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 203/421] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:20.278660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.306791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:20.324548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae82e9a-a19d-4bf1-a41e-66f80fa4ff19 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.381476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.382025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.382298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.382570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.394895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:20.400338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:20.401251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:20.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:20.402656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 219/422] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:20.407508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.464892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d90e38-2b35-440e-bd4f-774e49177575 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.493570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.494173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.494451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.494723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.563284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:20.572741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4fc5d-a2fb-452b-a56f-4ea8da425a5e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.587324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:20.588802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:20.610596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:20.657684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.658256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.658536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.658809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.659486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:20.659793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:20.661239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:20.685428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 220/423] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:20.693388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 204/424] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:58:20.693821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.698800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.797573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d556c875-9f6b-40ec-90fe-fc1e1c7f12ad None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.813196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:20.832575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.833249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.833525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.834170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.835490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:20.837497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:20.839298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 205/425] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:20.844856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:20.949672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9870d84-8c57-48f4-8356-95ec07922aea None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.958235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b964d71-ba71-4890-88de-65759e71ac41 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:20.985483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:20.986123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:20.986400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:20.986674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:20.988559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:20.990056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:20.994799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 206/426] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:21.003655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.037621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.038249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.038532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.038806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.057218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.065541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.110340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2155bf-1d83-4899-b38a-2be4ce8dda29 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.145476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.146352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.146620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.147490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=881f2ae0820241ec9d3216765931ba06) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:21.147803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.149431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.155896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 207/427] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:21.167556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.169084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.302466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 221/428] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 609 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:21.307941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.460513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214c67b5-f896-47fd-b073-387d93e1e89f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.477688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.501102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a320ef-ac8f-4d99-927b-66633cd0e8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.501743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a320ef-ac8f-4d99-927b-66633cd0e8fa None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.502025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a320ef-ac8f-4d99-927b-66633cd0e8fa None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.502296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a320ef-ac8f-4d99-927b-66633cd0e8fa None admin] PATH_INFO: `/v3/projects/881f2ae0820241ec9d3216765931ba06` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.503423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a320ef-ac8f-4d99-927b-66633cd0e8fa None admin] RBAC: Authorizing `identity:get_project(project_id=881f2ae0820241ec9d3216765931ba06)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.504852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a320ef-ac8f-4d99-927b-66633cd0e8fa None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.506819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 208/429] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/projects/881f2ae0820241ec9d3216765931ba06 => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:21.512303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.551598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9623774e-683b-4ad3-979f-e34e63bf35c9 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.551922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9623774e-683b-4ad3-979f-e34e63bf35c9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.552140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9623774e-683b-4ad3-979f-e34e63bf35c9 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.553685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 209/430] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:21.559458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.559740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.560010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.566982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.590488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:21.592001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:21.637270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b1f903-3b0c-437c-9bda-0e387c5a173d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.684711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 210/431] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:21.687550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcf5ebb-2d64-4e73-b912-518e433b093d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.688431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf5ebb-2d64-4e73-b912-518e433b093d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.688939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf5ebb-2d64-4e73-b912-518e433b093d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.689459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf5ebb-2d64-4e73-b912-518e433b093d None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.691178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcf5ebb-2d64-4e73-b912-518e433b093d None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.692823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcf5ebb-2d64-4e73-b912-518e433b093d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.694901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 222/432] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:21.701895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.803426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea9881-0669-4304-bdde-bee2c51dd777 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.815547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.839510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.840438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.840975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.841446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.842379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=881f2ae0820241ec9d3216765931ba06) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:21.842903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.844608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.850737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 211/433] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=881f2ae0820241ec9d3216765931ba06&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:21.995336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2118a1b2-8ebf-423a-8029-4bbaa073fe23 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.995690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2118a1b2-8ebf-423a-8029-4bbaa073fe23 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.995878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2118a1b2-8ebf-423a-8029-4bbaa073fe23 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.997558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 223/434] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:22.001747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.002854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.003158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.003401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.024131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:22.025446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:22.107697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 224/435] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:22.124331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07404c2-1462-4819-833e-f96ab9a18411 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.160098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.161169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.161688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.162238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.172970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:22.178484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:22.179381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:22.181220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:22.183689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 212/436] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:22.189839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.233975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.295165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93eb7ff-1744-4c66-8291-29bf162e661f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.334300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.334933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.335222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.335470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.336155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:22.336454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.338431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.348425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 225/437] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:22.354815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.378822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.396698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.485597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0b252-bad9-4cea-a8ff-e0f1c3ce85b6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.500528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed92226-756f-45b9-a0b4-970619cbe0a5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.502091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.528496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b216192-1618-4adf-a191-159efc07b10f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.529136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b216192-1618-4adf-a191-159efc07b10f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.529418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b216192-1618-4adf-a191-159efc07b10f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.529686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b216192-1618-4adf-a191-159efc07b10f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.536760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f756eff-e391-43b2-9c71-1c0965ed69b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.537420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f756eff-e391-43b2-9c71-1c0965ed69b1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.537700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f756eff-e391-43b2-9c71-1c0965ed69b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.537975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f756eff-e391-43b2-9c71-1c0965ed69b1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.548380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f756eff-e391-43b2-9c71-1c0965ed69b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-6f756eff-e391-43b2-9c71-1c0965ed69b1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:22.553587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:22.554464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:22.555251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 226/438] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:22.556818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bacccf3-844f-42b4-9dea-0afc35934c26 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.557169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bacccf3-844f-42b4-9dea-0afc35934c26 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.557377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bacccf3-844f-42b4-9dea-0afc35934c26 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.558750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 227/439] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:22.560830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bacccf3-844f-42b4-9dea-0afc35934c26 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.598073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b216192-1618-4adf-a191-159efc07b10f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.599509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b216192-1618-4adf-a191-159efc07b10f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.631504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.661038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bacccf3-844f-42b4-9dea-0afc35934c26 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.693870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.694468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.694741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.695007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.695655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:22.695952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.697288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 213/440] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:22.697887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.698592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.698888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.699290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.709538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 228/441] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:22.710502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.717195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:22.718403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:22.737578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.782323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 214/442] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:22.788931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.812920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ccb519-fa3f-490b-afc7-9e806c8631a2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.849608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.850230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.850510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.850784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.851960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.853470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.855349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.855461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 229/443] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:22.860541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.897939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81d8b4-5e87-4ba5-bf2c-b9312c4d67af None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.933936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.934545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.934828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.935103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.953213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.963651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5cfb41-07e5-4027-bc35-c03f7abad1d2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.997971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.998585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.998876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.999127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.000221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.001688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.003393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 230/444] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:23.006152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.007595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.008919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.073435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.110303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 215/445] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:23.111135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.118888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460e33bb-75a8-46be-9e63-5e883068a208 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.152159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.152674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.152981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.153253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.153954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:23.154254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.155676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.161172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 231/446] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 611 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:23.161924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.202693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.228172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44424867-d5bf-4f54-bd9c-04b8f2426cc7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.265467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.266113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.266395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.266667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.278230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:23.283236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:23.284165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:23.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 216/447] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:23.291016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.292850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423655fc-2a68-431f-98e8-426df45d6793 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.317857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.331821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.332466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.334060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.334060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] PATH_INFO: `/v3/projects/a7010d3512da4f9c9fab7d993427ebf8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.334225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] RBAC: Authorizing `identity:get_project(project_id=a7010d3512da4f9c9fab7d993427ebf8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.335953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.338781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 232/448] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/projects/a7010d3512da4f9c9fab7d993427ebf8 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:23.343481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.405416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169ee4f9-c815-4108-9c9d-b7cce9fa1b2a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.439851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.440967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.441245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.441515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.442194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:23.442493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.443605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.443856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.452155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 217/449] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:23.453203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6260d758-3b28-400a-a8fc-dc3bc2b88320 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.457773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.486192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.486814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.487090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.487359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] PATH_INFO: `/v3/groups/cabedb4b689143318780aebce32925c9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.488467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] RBAC: Authorizing `identity:get_group(group_id=cabedb4b689143318780aebce32925c9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.489923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.491607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 233/450] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/groups/cabedb4b689143318780aebce32925c9 => generated 226 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:23.497106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.540322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.555211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d379acf-badd-4fb9-b484-7635b317a0ea None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.590923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.591534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.591632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606e0036-b633-4681-bf9f-2347bebc5845 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.591986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.592126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.624224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.624814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.625114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.625382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.626083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] RBAC: Adding query filter params (group.id=cabedb4b689143318780aebce32925c9, role.id=ce237b90aca34abead4f05dfc46a1ec4, scope.project.id=a7010d3512da4f9c9fab7d993427ebf8) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:23.626388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.627804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.633390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 234/451] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/role_assignments?group.id=cabedb4b689143318780aebce32925c9&scope.project.id=a7010d3512da4f9c9fab7d993427ebf8&role.id=ce237b90aca34abead4f05dfc46a1ec4 => generated 611 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:23.660066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.661544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.749335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.765615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 218/452] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:58:23.771338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:23.798374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:23.876106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822c7970-67d7-4e0f-b186-410183334edb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:23.908819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:23.909469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:23.909745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:23.910011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] PATH_INFO: `/v3/projects/304c048daacb48678f2d64fc933b378e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:23.911150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] RBAC: Authorizing `identity:get_project(project_id=304c048daacb48678f2d64fc933b378e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:23.912535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:23.914320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 235/453] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/projects/304c048daacb48678f2d64fc933b378e => generated 344 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:23.920355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:24.016991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485370eb-37de-40f9-b99d-5c9a0ac38af3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:24.050973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30bc7e4-0393-4c5e-8bf9-324774f40967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:24.051847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30bc7e4-0393-4c5e-8bf9-324774f40967 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.052120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30bc7e4-0393-4c5e-8bf9-324774f40967 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.052496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30bc7e4-0393-4c5e-8bf9-324774f40967 None admin] PATH_INFO: `/v3/groups/39e94a1433754d898647d01c6e0c1376` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.053662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30bc7e4-0393-4c5e-8bf9-324774f40967 None admin] RBAC: Authorizing `identity:get_group(group_id=39e94a1433754d898647d01c6e0c1376)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:24.055106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30bc7e4-0393-4c5e-8bf9-324774f40967 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:24.056937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 219/454] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Sep 29 01:58:23 2022] GET /identity/v3/groups/39e94a1433754d898647d01c6e0c1376 => generated 229 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:24.062723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:24.154875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:24.163813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf618c04-68f5-4273-9d8e-2440b6d539c6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:24.196740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:24.197354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.197629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.197897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.198603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] RBAC: Adding query filter params (group.id=39e94a1433754d898647d01c6e0c1376, role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=304c048daacb48678f2d64fc933b378e) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:24.198906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:24.200344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9567b7-f5ec-4f96-b98e-b24a0c83ebb0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:24.205789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 236/455] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Sep 29 01:58:24 2022] GET /identity/v3/role_assignments?group.id=39e94a1433754d898647d01c6e0c1376&scope.project.id=304c048daacb48678f2d64fc933b378e&role.id=619e79e7bb94488780bd4da040d61c34 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:26.902957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf50d09-90a3-452f-aecf-e2b801a86fbc None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.903944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf50d09-90a3-452f-aecf-e2b801a86fbc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.904448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf50d09-90a3-452f-aecf-e2b801a86fbc None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.906315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 220/456] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:26.911306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.911717 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.911854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.936032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:26.937270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:26.977015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12375fce-f6f5-4608-b6de-96e8c1c46972 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.977323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12375fce-f6f5-4608-b6de-96e8c1c46972 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.977636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12375fce-f6f5-4608-b6de-96e8c1c46972 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.979148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 221/457] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:26.985174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9363a57-aafd-472a-930d-4bd12b4d848a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.985625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9363a57-aafd-472a-930d-4bd12b4d848a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.986045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9363a57-aafd-472a-930d-4bd12b4d848a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.017693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c9363a57-aafd-472a-930d-4bd12b4d848a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:27.019162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c9363a57-aafd-472a-930d-4bd12b4d848a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:27.049430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 237/458] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:27.106944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:27.131385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 222/459] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:27.377702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:27.491785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:27.512757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb6880e-6310-45d2-ba13-fdaa5fae2391 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:27.528535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929e477e-6bab-4904-b0ae-f5a16a7ce9e0 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.528823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929e477e-6bab-4904-b0ae-f5a16a7ce9e0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.529133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929e477e-6bab-4904-b0ae-f5a16a7ce9e0 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.530800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 223/460] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:27.537818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b94941d-cb23-4cfd-b84f-2887e4502fde None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.538106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b94941d-cb23-4cfd-b84f-2887e4502fde None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.538382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b94941d-cb23-4cfd-b84f-2887e4502fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.572922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1b94941d-cb23-4cfd-b84f-2887e4502fde None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:27.572922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1b94941d-cb23-4cfd-b84f-2887e4502fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:27.575932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:27.576575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.576921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.577199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.595783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:27.597286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-5630004a-6ec3-4c03-a123-e1d2ca475daa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:27.624690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:27.625680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:27.626531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:27.626531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:27.626531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:27.633396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 238/461] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:27.634656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496771b9-3340-4e9f-b89d-e2153f5e7ad9 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.634941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496771b9-3340-4e9f-b89d-e2153f5e7ad9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.635216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496771b9-3340-4e9f-b89d-e2153f5e7ad9 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.636674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 239/462] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:27.641307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496771b9-3340-4e9f-b89d-e2153f5e7ad9 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:27.680891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 224/463] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:27.682310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.682601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.682902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.704954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:27.706157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:27.753558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496771b9-3340-4e9f-b89d-e2153f5e7ad9 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:27.787569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:27.788126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.788405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.788676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.789346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:27.789646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:27.791640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:27.794566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 225/464] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:27.795667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:27.806876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:27.820959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 240/465] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:27.830532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:27.858943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:27.917664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43939cd-213c-459b-a987-5113fab902e6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:27.955698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:27.956658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.957351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.957821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.970461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:27.972620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:27.993275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:27.994234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:27.995062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:27.995062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:27.995062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:27.997330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 226/466] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:27.998491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.037419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158895a1-5aca-4464-b953-24b4603bd454 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.083965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.084581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.084861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.085160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.085835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.088164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.104191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:28.111287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d485c9cd-3037-4cc3-a4f6-333b43e9f72f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.111690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 241/467] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:27 2022] POST /identity/v3/users => generated 246 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:28.112513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.162264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.162988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.163267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.163542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.177330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.178735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.183235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-faa7179c-522f-42aa-9c21-48fd143b08e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:28.197700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:28.203273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 227/468] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d353318-6909-425d-9602-b77e252ac2f9 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d353318-6909-425d-9602-b77e252ac2f9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d353318-6909-425d-9602-b77e252ac2f9 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 228/469] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:28.204235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d353318-6909-425d-9602-b77e252ac2f9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.241457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929989f1-bbc0-41d3-ae55-b7db479e44c5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.278115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.278736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.279014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.279287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.279938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:28.280238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.281660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.295510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 242/470] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:28.296438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.334142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d353318-6909-425d-9602-b77e252ac2f9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.372756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.373409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.373689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.373957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.384682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.386147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.394284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-ab35228d-a763-4e82-957a-ae3e7b9bee96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:28.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:28.402896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:28.403851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:28.403851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:28.403851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:28.403851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 229/471] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:28.404821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.405132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.405404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.407887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a708cc45-9df5-4b10-84e6-58dda1de07cb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.422954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:28.424121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:28.425095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:28.452014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.452936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.453645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.454104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.455017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:28.455502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.457448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.469988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 243/472] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:28.471008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.504893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 230/473] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:28.504893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.599800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d1c8c-9131-431b-be69-23ef203f2980 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.610330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d73477-b562-4a12-b519-16128e8efce0 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.642429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:28.648440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.649073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.649369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.649675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.650329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.651892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.652425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.653265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.653731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.654207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.655112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:28.655604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.657195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.670299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 231/474] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:28.671695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.676256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 244/475] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:28 2022] POST /identity/v3/users => generated 248 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:28.677001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.722545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:28.823319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4419b1e0-961f-4f2c-9578-8bcf53d7b2eb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.837315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55043cb7-eadf-4558-a86e-2b3644ce1e77 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:28.874263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.875189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.875469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.875744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.876399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.877895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.880794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:28.881664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:28.882143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:28.882624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:28.883474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:28.885179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:28.912912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 245/476] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:28 2022] POST /identity/v3/users => generated 249 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:28.912912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:28.920409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 232/477] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:28 2022] POST /identity/v3/users => generated 248 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:29.029319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:29.049852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146e5774-3cc0-4e3e-8247-d441806f2999 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:29.091829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:29.092470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.092749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.093068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.093784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:29.095261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:29.106254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.sql.core [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 29 01:58:29.106254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 29 01:58:29.106254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [parameters: {'id': '19e7263c46ef4a4eb1a8018545f506af', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 29 01:58:29.106254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81544) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result.read() Sep 29 01:58:29.124007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return ref.to_dict() Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:58:29.125899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application yield resource Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.session.rollback() Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.force_reraise() Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise self.value Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application session.commit() Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 29 01:58:29.128071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._prepare_impl() Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.session.flush() Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._flush(objects) Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application compat.raise_( Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application flush_context.execute() Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rec.execute(self) Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 29 01:58:29.130173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:58:29.132360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result.read() Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:58:29.133839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [parameters: {'id': '19e7263c46ef4a4eb1a8018545f506af', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:29.138108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 29 01:58:29.140049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:29.142055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 246/478] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 29 01:58:28 2022] POST /identity/v3/roles => generated 144 bytes in 218 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:29.142198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:29.188396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:29.248213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd8ddab-1e60-4880-a5a8-759a83eccedb None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:29.285760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aeed63-8a80-4187-bcb7-4eb444786aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:29.286321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aeed63-8a80-4187-bcb7-4eb444786aca None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.286607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aeed63-8a80-4187-bcb7-4eb444786aca None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.286874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aeed63-8a80-4187-bcb7-4eb444786aca None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.298322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aeed63-8a80-4187-bcb7-4eb444786aca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-43aeed63-8a80-4187-bcb7-4eb444786aca None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:29.303484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:29.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:29.305293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 233/479] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:29.313363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:29.430722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d622fc21-a2cd-4597-a30b-2baafb5c0509 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:29.472728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:29.473359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.473632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.473900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.474553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:29.474848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:29.476209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7355defc-8c7b-4e94-a47d-c020627393ae None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:29.488739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 247/480] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:30.532355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda98240-6cb1-4ca6-a0fa-2d1dacdbcf68 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.532913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda98240-6cb1-4ca6-a0fa-2d1dacdbcf68 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.532913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda98240-6cb1-4ca6-a0fa-2d1dacdbcf68 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.534240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 234/481] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:30.539816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.540276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.540464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.570598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:30.571944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:30.580590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:30.666513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 248/482] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:30.991770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda98240-6cb1-4ca6-a0fa-2d1dacdbcf68 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:31.105303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda98240-6cb1-4ca6-a0fa-2d1dacdbcf68 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:31.113481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:31.149073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:31.150229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.150515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.150785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.161314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:31.166089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:31.166979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:31.167777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 235/483] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:31.173981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:31.315052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c364bd23-887c-4070-a2c4-9d63972c55f8 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:31.361803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:31.362414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.362696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.362963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.363624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:31.363920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:31.365328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb6636d-4b21-477e-b3e5-a45586ad033e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:31.376612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 249/484] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:31.382352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:31.510419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51bba0e-91c5-41ed-bf01-1745dfa4437c None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.510755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51bba0e-91c5-41ed-bf01-1745dfa4437c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.510957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51bba0e-91c5-41ed-bf01-1745dfa4437c None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.512529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 250/485] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:31.522042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde1990d-c8be-487e-afa3-e55d84c57cd6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.522320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde1990d-c8be-487e-afa3-e55d84c57cd6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.522586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde1990d-c8be-487e-afa3-e55d84c57cd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.524644 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52930012-ca87-4bc7-a394-fb813e8ff442 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:31.546974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:31.563456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b975431f-2986-4930-9294-e3993835c686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:31.564084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b975431f-2986-4930-9294-e3993835c686 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.564372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b975431f-2986-4930-9294-e3993835c686 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.564648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b975431f-2986-4930-9294-e3993835c686 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.575944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b975431f-2986-4930-9294-e3993835c686 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:31.577387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b975431f-2986-4930-9294-e3993835c686 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:31.577862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-bde1990d-c8be-487e-afa3-e55d84c57cd6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:31.579062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-bde1990d-c8be-487e-afa3-e55d84c57cd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-b975431f-2986-4930-9294-e3993835c686 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:31.592269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:31.593226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:31.594065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:31.594065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:31.594065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:31.594065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 236/486] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:31.601993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b975431f-2986-4930-9294-e3993835c686 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:31.637270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:31.735272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 251/487] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:31.771470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b975431f-2986-4930-9294-e3993835c686 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:31.808099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:31.808760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.809369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.809645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.810338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:31.810642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:31.812014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:31.817700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f52a09-17ff-4e29-9517-8ecd44e2d9a4 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.817982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f52a09-17ff-4e29-9517-8ecd44e2d9a4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.818256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f52a09-17ff-4e29-9517-8ecd44e2d9a4 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.819653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 252/488] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:31.830073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca60a3f-591c-49a6-979e-07faca04c3b9 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.830302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca60a3f-591c-49a6-979e-07faca04c3b9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.830549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca60a3f-591c-49a6-979e-07faca04c3b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.838658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 237/489] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:31.844097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:31.860242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fca60a3f-591c-49a6-979e-07faca04c3b9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:31.861507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fca60a3f-591c-49a6-979e-07faca04c3b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:31.962503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:31.968952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 253/490] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:31.970267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79110771-efee-42cb-91f1-5d57d28c9c71 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.970552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79110771-efee-42cb-91f1-5d57d28c9c71 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.970885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79110771-efee-42cb-91f1-5d57d28c9c71 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:31.972302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 254/491] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:31.986542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:31.986811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:31.987086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.008582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559cf98d-ef4d-4229-9b63-72b689d05abf None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.015029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:32.016224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:32.051008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.051638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.051972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.052246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.060693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.062177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:32.076606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:32.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:32.078580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:32.082348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 238/492] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:58:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:32.083135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.107926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.116190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 255/493] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 29 01:58:32.117403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.209681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21286290-0c05-4995-a9e8-d7487d38800e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.229791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbabf83-3397-4a71-976c-496ef433b36a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.266725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.267338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.267615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.267883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.268609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:32.269002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.270453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.277561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.278193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.278467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.278733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.290656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.297998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:32.303096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:32.303975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:32.304758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:32.307565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 239/494] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:32.308596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.309746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 256/495] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:32.314070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.378341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.432825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1e970b-eacd-4271-97b8-77fa77f6f823 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.433429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935e480a-d16b-400b-a017-0066d733e7e2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.473235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.473865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.474143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.474411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.474714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.475313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.475590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.475859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.476515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:32.476814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.478756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.486316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:32.488975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 257/496] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:32.488975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:32.491600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:32.492648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 240/497] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:32.493580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.555183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.596847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02b6d0-7e23-456e-bb76-70816c0c4818 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.598810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34847fc-f3b2-49a7-abd8-d6fc64afbcd5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.610377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.631585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.631733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.632122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.632243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.632403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.632499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.632669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.632754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.642829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:32.648007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:32.649097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 241/498] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:32.650450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.703773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.705268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.747088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1079ac-dbfc-425c-8065-4eb0b27437d8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.763532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.782052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.782670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.782945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.783213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.794029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.795463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.801080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 258/499] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:32.801910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7a495061-2843-4bf1-88eb-931cfdff3839 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:32.813629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:32.814796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:32.816058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:32.816058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:32.816058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:32.817123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 242/500] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:32.817368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.818307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cfe01d-9a12-4957-8d0a-e1682e9d4713 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.818462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cfe01d-9a12-4957-8d0a-e1682e9d4713 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.818744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cfe01d-9a12-4957-8d0a-e1682e9d4713 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.820915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 243/501] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:32.820915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cfe01d-9a12-4957-8d0a-e1682e9d4713 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.901148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f6160-3490-4e28-8290-f37875ccc984 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.921366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cfe01d-9a12-4957-8d0a-e1682e9d4713 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.933899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.934490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.934781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.935062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.935715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:32.936012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.937498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.945426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 259/502] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:32.946194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.956272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.956903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.957189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.957462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.958114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:32.958417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.959762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.968650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 244/503] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:32.969494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.998652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.023481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.046372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677e49d-0a1d-4187-b08d-0228172a71ed None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.065222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea292c99-f2b6-479b-b02c-8fd4cd805b91 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.078718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.079340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.079615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.079882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.090791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.092244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.098176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.098771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.099046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.099312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.099934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:33.100232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.101586 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-6e4d8cf0-1552-4bad-a493-6c8a0a1c84c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:33.106242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:33.107290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:33.108136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:33.108136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.108136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:33.108136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 260/504] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:33.108404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.108677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.108985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.116811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 245/505] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:33.117655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.126589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:33.127775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:33.194973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 261/506] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:33.195729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.219889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.230776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381ca035-8f7e-4506-997c-8a5f51e04c95 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.258260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.266940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.267831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.268175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.268506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.279096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.280526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:33.295375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:33.296273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:33.297126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:33.297126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.297126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:33.297126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 246/507] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:33.297352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.308397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c823faa-a21c-4c49-8497-584105b3e1c9 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.343904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.344533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.344809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.345110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.356263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.357824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:33.373007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:33.373922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:33.374768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:33.374768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.374768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:33.374768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 262/508] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:33.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.401295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd65df-9793-41cb-bdf3-c85b2c3a9366 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.438370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.438971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.439246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.439506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.440130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:33.440426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.442061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.459468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 247/509] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:33.459468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.461745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.489246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742fa967-d608-4c5b-acdc-869ec38f5ea5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.500130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.523551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.524376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.524659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.524964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.532470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.534001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:33.546234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:33.547358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 263/510] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/users/glance => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:33.548407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.564230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4c65a3-3d6d-4b8b-9ddc-2e33545e9462 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.604214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.604816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.605124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.605394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.606022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:33.606319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.607668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.619365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 248/511] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:33.624187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.645718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aef038-4ac1-462a-844a-d24d13ad1c18 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.678599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.679207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.679488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.679793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.680429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:33.680735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.681988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.682220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.693975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 264/512] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:33.693975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.721648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06cbab-d460-4a42-85e6-251c296477be None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.724350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.754520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.755139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.755418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.755686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.762625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.764058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:33.776431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:33.777614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:33.778940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 249/513] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:33.783452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.799975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb67b20-2c2d-4c77-ac82-db641d5632b1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.838526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.839145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.839486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.839856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.850564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.852033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:33.866328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:33.867389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:33.868297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:33.868297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:33.868297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:33.872422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 265/514] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:33.873249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.880188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0f29f-e87d-4acf-9bec-6efc3225da23 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.906190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.918093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.918697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.918971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.919239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.919951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:33.920276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.921718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.954871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:33.958908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 250/515] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:33.959554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:33.975584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d195e7-c331-4dfc-9c3a-559afa01e9f9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.009474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.010095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.010372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.010641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.017848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.019261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:34.030287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:34.031167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:58:34.032097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:34.033289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 266/516] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/users/cinder => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:34.034285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.062001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70514f37-6062-41d1-9ef6-f30e85da3966 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.095119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.095740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.096034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.096302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.103531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.104994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:34.116245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:34.117197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:34.118085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 251/517] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:33 2022] GET /identity/v3/users/neutron => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:34.119208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.137184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4834f3dc-8da6-416e-a372-58f2b89104d3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.166325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.171605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.172279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.172621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.172984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.173738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:34.174111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.175535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.194292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 267/518] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:34.195174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.201831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.222993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729971ac-b1d9-4c93-825b-f67bb2785fdc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.256811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.257472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.257712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.257986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.258618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:34.258947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.260316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.271798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 252/519] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:34.272494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.299081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d88bdf-f2b7-4462-84ed-9e53ffd25db7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.332159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.332762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.333061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.333328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.369267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e492f30a-a155-4cba-889f-a7c6aa8b4ade None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.370739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.398645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.400685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.402519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.404250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.404842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.405149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.405425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.406141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:34.406448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.407853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.424317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 253/520] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:34.425225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.511971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 268/521] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:34.512840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.524892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6be82bf-9f3d-4c75-b123-d8730217df7b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.559481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.560090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.560368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.560655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.561410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:34.561715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.563143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.583405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 254/522] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:34.583570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.604107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.612031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f574b7-0372-44e6-a9aa-0cc72d04e916 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.613557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.645797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.646391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.646641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.646908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.647612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=e528956ee4704ac0b5ce4c62826b71ee) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:34.647913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.649434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.654514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 269/523] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/role_assignments?user.id=e528956ee4704ac0b5ce4c62826b71ee&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:34.655280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.694773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f62c29-87fb-41d9-b07b-444b66f963c9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.731443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.732072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.732352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.732641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.733299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.734761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.769158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 255/524] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:34 2022] POST /identity/v3/users => generated 247 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:34.770441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:34.770712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97b34ca-3898-4894-9159-8b7337833680 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.827591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.828224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.828505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.829066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.894379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.906714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:34.926938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8438c4e7-6a99-4a07-bd37-47580fc805dc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:34.954792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.957036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:34.976954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:34.978153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:34.978442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:34.978714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:34.994059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:34.995584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:35.015583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:35.016679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:35.017546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:35.017546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:35.017546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:35.017546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 256/525] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:35.017761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.084314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 270/526] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:35.085515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.148515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d707fa31-7c88-4ed7-8e96-6729c6755fcc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.160849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.189195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.199772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.200386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.200666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.200999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.208308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb3e31-6a3c-4988-9dd0-2addf01d53ab None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.250051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.250800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.251086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.251360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.252012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:35.252317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.253862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.267128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 271/527] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:35.267784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.283298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.285503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.381567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5570132e-15bb-4877-9ab4-c130d25260ca None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.413313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 257/528] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:35.413496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.414785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.424791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.425671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.426022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.426369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.438924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.441143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.450646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:35.458771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:35.459714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:35.460562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:35.460562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:35.460562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:35.463370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 272/529] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:35.464420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.549975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73893a0-54dd-4c70-829f-59d1610da7d1 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.581268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0bba62-9d4a-4040-b856-a2a35206d219 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.599429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.600073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.600352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.600622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.609436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.611152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:35.623766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:35.624666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:35.625920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 258/530] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:35.626383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.630049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.630959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.631456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.631986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.644724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.646667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:35.662912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:35.663843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:35.664695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:35.664695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:35.664695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:35.666883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 273/531] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:35.667991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.705322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.726621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.744797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a848524c-8064-4bff-afc1-7516079d78a3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.786537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.787159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.787438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.787710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.788358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:35.788658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.789496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f8b44d-a501-414d-a9d7-8eb538c70e0f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.790645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.807888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 259/532] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:35.807888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.831800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.832788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.833452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.833925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.834800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:35.835318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.837733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.851185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 274/533] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:35.852434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.912067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a188369d-90c0-4cab-bd89-1270ed339523 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.950063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:35.954746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:35.955823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:35.956106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:35.956381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:35.957111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:35.957346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:35.958811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:35.963194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7388005-fd36-47e4-a168-66a9e8a3a3d3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:35.972005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 260/534] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:35.972911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:35.982497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:36.002426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.003448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.004036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.004532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.012703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.014487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:36.028304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:36.029212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:36.030070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:36.032156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 275/535] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:36.033167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:36.086375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f60ecb-be55-4903-954a-dfe905a7e46b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.123204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.123929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.124191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.124469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.125227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=0298fb95ae9341c6a1eb8da546d93cb2) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:36.125870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.127113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.133646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 261/536] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:35 2022] GET /identity/v3/role_assignments?user.id=0298fb95ae9341c6a1eb8da546d93cb2&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:36.135088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:36.146923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a1f1c9-03d7-409f-9363-5e96d53c6360 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.185828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.186924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.187515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.187989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.197165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.198980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:36.212584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:36.213509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:36.214356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:36.215649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:36.217329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 276/537] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:36.229987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:36.249489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57aba11-65ac-498e-917b-cac00be40fc6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.252113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:36.289023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.290111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.290601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.291067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.292013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:36.292516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.294249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.329428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 262/538] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:36.335881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:36.374890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149c1c0-19f4-4854-8b25-58317bdcbc2e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.412726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.413383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.413665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.413941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.414642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:36.414982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.416442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.431165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 277/539] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:36.436724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:36.527266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858b870a-3d46-4889-b889-221da670dff9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.560179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:36.566219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.567117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.567594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.568080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.569051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=09c034ad1ccd45e78a4b3aeab2129227) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:36.569585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.571335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba153a73-47b8-4ba1-9165-2caadae08257 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.584501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 263/540] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:36 2022] GET /identity/v3/role_assignments?user.id=09c034ad1ccd45e78a4b3aeab2129227&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:36.613982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:36.665029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420886cc-a89b-448b-9ef3-d4ac941f970f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.721802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.722421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.722705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.722973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.723766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=afdc21c25f734a51aa520e7d4e53455b) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:36.724074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:36.725691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a647aa3-2aed-41a3-8723-da37cf24805d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:36.733091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 278/541] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:36 2022] GET /identity/v3/role_assignments?user.id=afdc21c25f734a51aa520e7d4e53455b&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:37.094496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffde48b-2ba3-4cf2-86e2-6a9c53429f00 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.095032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffde48b-2ba3-4cf2-86e2-6a9c53429f00 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.095032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffde48b-2ba3-4cf2-86e2-6a9c53429f00 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.096743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 264/542] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:37.102083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c60d4c-1fad-4b4f-aa24-67bdbb56c996 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.102367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c60d4c-1fad-4b4f-aa24-67bdbb56c996 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.102680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c60d4c-1fad-4b4f-aa24-67bdbb56c996 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.129983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-92c60d4c-1fad-4b4f-aa24-67bdbb56c996 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:37.131198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-92c60d4c-1fad-4b4f-aa24-67bdbb56c996 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:37.232913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 279/543] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:37.263650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff00343-deba-4112-ad5d-7899b13f07d1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.263948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff00343-deba-4112-ad5d-7899b13f07d1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.264205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff00343-deba-4112-ad5d-7899b13f07d1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.265778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 265/544] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:37.271311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.271601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.271876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.298653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:37.299860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:37.384276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.395106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 280/545] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:37.667205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff00343-deba-4112-ad5d-7899b13f07d1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.719680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.814567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff00343-deba-4112-ad5d-7899b13f07d1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.883328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.884533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.885887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.886984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.905869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:37.912314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:37.913218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:37.913995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:37.920952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 266/546] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:37.925630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da95410-6d79-4877-8832-2850b3edc8bb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.925748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.987311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.987947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.988222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.988486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.014232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:38.017205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:38.019923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:38.020803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:38.022490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 281/547] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:38.028512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.059746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1c2c49-b3ef-4aa0-97d8-e34f08819cef None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.097030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.097639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.097899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.098170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.098826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:38.099125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.100489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.108816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 267/548] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.115875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.133254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.207148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02a7621-a264-4c54-9947-bfe35b3abe71 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.253594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2f7649-7293-4ce3-81b3-f16749fa87a9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.271876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.293327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.294634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.295479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.296213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.297579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:38.298364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.299143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.299552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.299962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.300415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.302875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.320923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 282/549] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:38.323491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.325134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.327310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:38.354678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:38.355997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:38.357069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:38.357069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:38.357069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:38.365751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 268/550] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:38.371792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.524663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7105d993-057d-41ea-af8e-3e8396843de3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.541796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d26d85c-8f12-41d1-844f-f427b660058b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.563557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.583763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.584650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.585097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.586045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.597311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.599160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:38.615527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:38.616715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:38.617986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:38.617986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:38.617986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:38.619862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 283/551] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:38.624444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.625855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.626885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.627693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.631221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.633864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:38.634179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.635612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.664716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 269/552] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:38.670207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.701927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.739861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d1bbd4-32d6-41c7-ae4f-5b0e668f37f1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.789699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.790756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.791295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.791784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.792628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:38.793163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.794761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8825f5-60a5-46e0-8308-edf913dc9889 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.811610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d6b1b-4a91-40e0-a001-9e9b55ac20f6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.816554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 284/553] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.818114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e42d053-96b5-48a8-af14-4067847c68ea None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.818603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e42d053-96b5-48a8-af14-4067847c68ea None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.819145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e42d053-96b5-48a8-af14-4067847c68ea None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.820523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 285/554] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:38.825027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e42d053-96b5-48a8-af14-4067847c68ea None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.861104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.861884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.863404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.863688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.863965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.872678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.874179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-81d7b290-90ef-420a-a6ac-80bc99a3a4c6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:38.887637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:38.888574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:38.889498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 270/555] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:38.892890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.892890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.892890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.912332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:38.913776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:38.957320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e42d053-96b5-48a8-af14-4067847c68ea None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.994660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.995356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.995690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.995964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.001938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 271/556] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:39.002799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.010864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.012454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.023168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:39.030866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:39.032328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:39.034116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:39.040956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 286/557] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:39.040956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.122320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d4713e-81c1-4904-8025-0838ac602f63 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.162407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.163017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.163297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.163568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.164295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:39.164598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.167435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.196811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 272/558] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.198459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.220337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.245156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04cd025-1bd1-4021-a812-fe8bdfc467c8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.312102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.312625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.312915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.313190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.313913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:39.314214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.315895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17abf4cc-46c8-4308-a07c-39fba92f731e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.337695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.345687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 287/559] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.346767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0e0bf-56cf-4bac-b3f3-ce2930ec8cc3 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.347042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0e0bf-56cf-4bac-b3f3-ce2930ec8cc3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.347314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0e0bf-56cf-4bac-b3f3-ce2930ec8cc3 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.352277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 288/560] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:39.352277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72089611-5bc2-43d5-b0f9-ac35dbdaff92 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.352277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72089611-5bc2-43d5-b0f9-ac35dbdaff92 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.352277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72089611-5bc2-43d5-b0f9-ac35dbdaff92 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.352277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 289/561] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:39 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:39.352277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72089611-5bc2-43d5-b0f9-ac35dbdaff92 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.370402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad61e0d1-f4f5-4aed-b3d9-ad933e8fa669 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.404986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.405608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.405886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.406158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.416399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.417867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-e8485207-d7e8-46ab-a358-31a3645c16ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:39.432549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:39.433726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:39.434582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:39.434582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:39.434582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:39.434582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 273/562] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:39.434809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35618b24-a7c9-4641-9641-65a6f5f146c9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.435062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35618b24-a7c9-4641-9641-65a6f5f146c9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.435492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35618b24-a7c9-4641-9641-65a6f5f146c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.452963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-35618b24-a7c9-4641-9641-65a6f5f146c9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:39.454149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-35618b24-a7c9-4641-9641-65a6f5f146c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:39.458834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72089611-5bc2-43d5-b0f9-ac35dbdaff92 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.495840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.496455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.496737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.497069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.522260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.533842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 274/563] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:39.536005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.536300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.536738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.559334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:39.560536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:39.579779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.583954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.637324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.652535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 275/564] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:39.661869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.756491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 290/565] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:39.757389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.774997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611d88be-9815-4a20-8b33-1359c4142485 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.822868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.824539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.824894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.825170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.831789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.839626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:39.844641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:39.845566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:39.846397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 276/566] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:39.847075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.904827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.924236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bf58ef-6ce1-46a2-b361-58c9753db373 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.968981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.969596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.969877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.970147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.970349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9fb8b0-5399-49d6-8e9b-6efcf7ac70b0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.970786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:39.971092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.972652 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.985625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 291/567] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.986444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.005395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.006017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.006293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.006569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.016743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.018245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:40.032464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.033612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:40.034704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:40.034704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:40.034704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:40.034704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 277/568] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.034704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.087278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaecd81-cc8f-4b23-bc72-64bb2a1fd4b5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.093804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.120233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.120985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.121268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.121543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.128938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.131323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:40.131933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186a3a17-0396-4870-b998-1b74728104d7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:40.137261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.138270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 292/569] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.139106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.166796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.167393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.167670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.167941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.168595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.168913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.170276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.178084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 278/570] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:40.178924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.237698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f79102-c302-44f3-bc79-8840bb264a3f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.269857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.270479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.270753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.271028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.276861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd998bc-746a-48af-96aa-88389c2d2d34 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.281127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:40.286207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.287099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:40.287927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 293/571] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.288608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.314088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.314829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.315495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.316027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.316302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.323357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.324776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:40.336237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.337174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:40.338100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 279/572] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.338587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.361463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.388223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c55cd9-d1bf-4f01-94ef-028553878509 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.421202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.421809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.422132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.422410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.423049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.423358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.424744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.434969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821aae9c-e91d-4852-aa54-b87581a54d82 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.436504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 294/573] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:40.437332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.468452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.469086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.469369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.469644 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.470306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.470612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.471958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.479629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 280/574] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:40.480409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.519633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.534130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cd0ad3-8d6e-4db2-a66b-b3e0989cd920 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.562543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.570979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.572137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.572416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.572687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.579232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957dbfef-30b8-4e68-99fc-e0f69cc9520a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.582712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.584178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:40.599057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.600201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:40.601454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:40.601454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:40.601454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:40.605187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 295/575] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.605893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.615506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.616109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.616387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.616659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.617349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.617652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.619044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.626706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 281/576] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:40.629979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.702164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ba30a2-b6fb-4388-be3a-aa2fa59119db None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.730363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7179353b-6183-479e-a987-bd53a550d660 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.739294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.739903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.740177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.740449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.741164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.741473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.742915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.751774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.758147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 296/577] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:40.759449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.763949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.764635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.764936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.765216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.771670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.773137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:40.784585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.785666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 282/578] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.791574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.798012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:40.858548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c286354-1a56-4e2b-8fc0-fb847f1c2df6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.887555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b469677-3a4f-46ce-af4b-06a4a215c2ba None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.890645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.891259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.891556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.891825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.901921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.903344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:40.917241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:40.918335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:40.919299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:40.919299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:40.919299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:40.919299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 297/579] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:40.919563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.928466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.929099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.929374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.929653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.930458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.931294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.932659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.945177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 283/580] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:40.945971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.976661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.010618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.026176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07881121-7c7c-4b8c-b9f6-2c87bd417e8d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.044140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0f26b4-e3b6-48e6-8a1f-24c3689023d5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.058076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.058704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.058979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.059246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.069609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.071036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.077769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.078416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.078696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.078972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/e528956ee4704ac0b5ce4c62826b71ee/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:41.085037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:41.086159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:41.087137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:41.087137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:41.087137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:41.087137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 298/581] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:41.087137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.097934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=e528956ee4704ac0b5ce4c62826b71ee, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.099353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.111625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 284/582] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:40 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/e528956ee4704ac0b5ce4c62826b71ee/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:41.112347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.201061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f55355-ac08-4583-8904-0c43f88b4fe8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.202866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.230752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c27a078-843b-4371-a620-2f2b227313a5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.239537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.240234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.240525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.240798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.241545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:41.241853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.244317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.251303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.262793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 299/583] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.262793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.282696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.282696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.283009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.283245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.283862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:41.284160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.285543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.299235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 285/584] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.300043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.374326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97811024-2fa0-4b52-96a4-07dc40f4b560 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.409263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.409884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.410156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.410421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.410938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76b8255-beab-4f2a-b8d3-e69d9223ea5e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.418266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.419711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:41.432302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:41.433309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 300/585] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:41.434845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.450453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda265c9-719a-4108-a052-c54475562667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.451381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda265c9-719a-4108-a052-c54475562667 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.451663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda265c9-719a-4108-a052-c54475562667 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.452084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda265c9-719a-4108-a052-c54475562667 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.452727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda265c9-719a-4108-a052-c54475562667 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:41.453060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda265c9-719a-4108-a052-c54475562667 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.454420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda265c9-719a-4108-a052-c54475562667 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.463875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.466835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 286/586] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.467655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda265c9-719a-4108-a052-c54475562667 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.507074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.550663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c17d76-fcae-443a-88a8-469edb09a6f1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.584057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda265c9-719a-4108-a052-c54475562667 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.589404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.590083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.590422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.590754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.591547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=10649368308f4dc4b8b887319cbcac8f) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:41.591910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.593452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.600914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 301/587] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/role_assignments?user.id=10649368308f4dc4b8b887319cbcac8f&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.601815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.642749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.643736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.644186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.644633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.653913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.655823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:41.672227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:41.673270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:58:41.674209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:41.675503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 287/588] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/users/neutron => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:41.676606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.721549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.746567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5c2a3b-f07b-4f51-95df-962950359608 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.789390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.790076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.790832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.791109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.791838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:41.792172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.793588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.805249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.811573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 302/589] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.812219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.813452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d2bad1-f101-4aa3-a7a2-f990fca1acbc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.860767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.862303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.862789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.863745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.874526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.877440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:41.893464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:41.894408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:58:41.895322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:41.903463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 288/590] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/users/cinder => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:41.903463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.928534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f054a50b-cc12-497b-94a3-14eea9125c1c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:41.967364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:41.968042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.968321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.968595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.969354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:41.969660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:41.971226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:41.994887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 303/591] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.995743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:41.998496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.019109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5c569-2be0-461e-a06c-b21c05583d81 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.058648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.059946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.060427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.061297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.074657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.077721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:42.095489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:42.096453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:42.097373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:42.097373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:42.097373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:42.103143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 289/592] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:42.103143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.115863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.123359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ecc5cf-1b3f-46fa-aae2-f86b869f992f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.168153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.169397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.169899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.170575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.172164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:42.172671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.174575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.203545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 304/593] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:42.203545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.220444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da9b613-e3f2-4407-a644-1667bf2989e3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.256535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.257352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.257750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.258122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.270237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.271854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.284079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:42.288243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:42.289421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:42.291076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:42.291076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:42.291076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:42.292610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 290/594] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:42.293526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.339479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b46a01a-c271-4de1-930e-8024cd69a936 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.384295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.385460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.385991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.386503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.387248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:42.387810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.389606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.408986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 305/595] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:42.408986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.410173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece9e36-32a7-40d6-8929-314e439805cf None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.414348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.457440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.458589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.459052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.459511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.473854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.475915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:42.495634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:42.496825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:42.497945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:42.497945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:42.497945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:42.501093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 291/596] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:42.504663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.528453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55275bff-8ba5-4b28-95aa-0cfe17d280c1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.549595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.564782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.568271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.568271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.568271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.568271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:42.568271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.571437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.586094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 306/597] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:42.587577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.627212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd95e99-edc0-4720-91bc-ac734d19f1cf None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.641096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.665969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.666952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.667435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.667937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.675213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.676928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:42.689110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:42.689990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:42.691136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:42.691549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11856076-0172-4819-a6b4-3f167fa5f144 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.697890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 292/598] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:42.698942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.726525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.728014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.728014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.728477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.729345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:42.729723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.731178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.743634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 307/599] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:42.744681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.787708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.817525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb491c-7930-4763-8901-987c8d2ab1be None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.851127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386eaed2-fb64-4968-891f-9efba3535c51 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.855080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.855903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.856328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.856742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.864370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.866116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:42.879265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:42.880297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:42.881233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:42.883730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 293/600] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:42.884894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.889713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.889713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.890033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.890249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.891021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:42.891376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.892862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.907117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 308/601] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:42.907117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.926098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.004459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9a0c54-c64a-42e8-bebc-667f41b4b018 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.016397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.020538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f888418-39c4-43d1-a775-a6a9892aac8c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.042952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.043837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.044264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.044677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.052336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.054111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.057172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.057768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.058147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.058517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.059384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:43.059786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.061334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:43.067575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:43.068562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:43.069489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:43.075785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 309/603] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.076044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 294/603] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:43.076145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.078605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.173160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.180041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02076b6a-42e2-414c-917e-991e72b88325 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.184374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d581ba1b-a4b1-421b-bf90-6c9e369e2cdc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.219494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.220181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.220461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.220735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/0298fb95ae9341c6a1eb8da546d93cb2/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.221984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.222863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.223383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.223856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.224767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:43.225331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.227019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.238686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=0298fb95ae9341c6a1eb8da546d93cb2, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.240154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.247854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 295/604] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.249245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.260972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 310/605] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:43 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/0298fb95ae9341c6a1eb8da546d93cb2/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:43.260972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.289727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.379391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c5d694-77df-49a3-8b43-7b518206fa21 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.405360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e1cf60-acb4-49a8-bd50-1fed990f6581 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.420244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e73e6-8462-46ba-bd54-ec8801473e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.421082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e73e6-8462-46ba-bd54-ec8801473e99 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.421362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e73e6-8462-46ba-bd54-ec8801473e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.421639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e73e6-8462-46ba-bd54-ec8801473e99 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/09c034ad1ccd45e78a4b3aeab2129227/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.439517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e73e6-8462-46ba-bd54-ec8801473e99 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=09c034ad1ccd45e78a4b3aeab2129227, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.441674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0e73e6-8462-46ba-bd54-ec8801473e99 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.444085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c036236-4a8f-4dfb-b936-c8a8d3669dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.445196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c036236-4a8f-4dfb-b936-c8a8d3669dc3 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.445656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c036236-4a8f-4dfb-b936-c8a8d3669dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.446126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c036236-4a8f-4dfb-b936-c8a8d3669dc3 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/afdc21c25f734a51aa520e7d4e53455b/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.458586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 311/606] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:43 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/09c034ad1ccd45e78a4b3aeab2129227/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:43.460123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff259a6d-84aa-4b27-8bc5-64e22e98fe07 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.460681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff259a6d-84aa-4b27-8bc5-64e22e98fe07 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.461055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff259a6d-84aa-4b27-8bc5-64e22e98fe07 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.466767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 312/607] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:43.468716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83678016-bb86-4c80-b500-63e0b43fdf76 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.469143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83678016-bb86-4c80-b500-63e0b43fdf76 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.469487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83678016-bb86-4c80-b500-63e0b43fdf76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.470100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c036236-4a8f-4dfb-b936-c8a8d3669dc3 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=afdc21c25f734a51aa520e7d4e53455b, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.471832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c036236-4a8f-4dfb-b936-c8a8d3669dc3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.492546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-83678016-bb86-4c80-b500-63e0b43fdf76 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:43.494964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-83678016-bb86-4c80-b500-63e0b43fdf76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:43.501352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 296/608] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:43 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/afdc21c25f734a51aa520e7d4e53455b/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:43.591336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.607781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 313/609] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:43.837631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9517cbda-6a3b-4b45-98c3-fd4fe020e7a1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.837894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9517cbda-6a3b-4b45-98c3-fd4fe020e7a1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.838095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9517cbda-6a3b-4b45-98c3-fd4fe020e7a1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.839688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 297/610] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:43.845201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.845493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.845770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.864328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:43.865550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:43.935791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 314/611] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:43.945252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9517cbda-6a3b-4b45-98c3-fd4fe020e7a1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.948198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.063153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9517cbda-6a3b-4b45-98c3-fd4fe020e7a1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.109384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.109933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.110208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.110477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.123997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:44.128767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:44.129755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:44.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 298/612] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:44.135808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.269360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.289504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc669a13-8e59-4bf9-93e6-eca19ea9c0fd None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.343034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.357503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.389347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.389886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.390168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.390443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.391115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:44.391425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.392814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.409849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66aea97-5709-45e3-a086-dfbbf2b71ee6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.413713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 315/613] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.419824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.485010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.485645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.485926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.486430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.509307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:44.517467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:44.518399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:44.519224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:44.519847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 299/614] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:44.527994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.638528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd89e2a6-16aa-49b4-a67a-8caaf4a0dc0e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.678067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d384036-2f9d-4c1d-8917-3c9943a10f05 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.705044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.719202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.720707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.721474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.722442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.723188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.724440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.725057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.726468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.727308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:44.728011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.731569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.744526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.746173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.750968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 300/615] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.757268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:44.777928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:44.779123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:44.780228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:44.780228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:44.780228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:44.785471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 316/616] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 366 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:44.791308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.855561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.924439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651aefc3-4bfa-4363-9610-125d8d6a4709 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.964359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.964931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.965215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.965487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.984860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.986333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.988641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69949911-dfc2-4a87-8380-d6a70307de98 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:45.016005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:45.017038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:45.017934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:45.017934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:45.017934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:45.018388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 301/617] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:45.024575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.058736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.059352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.059630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.059899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.060545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:45.060841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.067249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.090519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 317/618] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:45.095793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.099137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.221821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a444c4-ccc0-4abf-a68d-c24b2b4c4d47 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.273255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.278133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f6e6ba-b59a-4dc4-8998-0adde7803b96 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.282455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.283071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.283352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.283626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.284325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:45.284626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.286068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.308128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 302/619] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:45.313433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.324889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.325582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.325913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.326184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.336373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.338179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:45.355948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:45.357000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:45.357978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:45.360955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 318/620] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:45.366415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.438584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.448194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9539070-dad5-4ad2-baa9-49bdbd0b3488 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.509186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.511848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.512366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.515270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.532101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.533896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:45.557560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:45.558588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:45.559565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:45.563189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 303/621] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:45.569160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.579895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f491e803-59e2-4e27-84ec-6af1be583921 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.633844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.634845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.635877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.636578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.637635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:45.638166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.639836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.662280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 319/622] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:45.669870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.695529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.775112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c24453-a262-4af5-a51c-f6cc99eb4bb1 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.815125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbc5a16-9511-4a10-97e4-2c5190a53450 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.840623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.841251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.841527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.841791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.842511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:45.842815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.844235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.875911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 304/623] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:45.882077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.885736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3c3135-20d3-4c97-b88f-8b951d0480b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.886355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c3135-20d3-4c97-b88f-8b951d0480b0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.886627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c3135-20d3-4c97-b88f-8b951d0480b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.886893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c3135-20d3-4c97-b88f-8b951d0480b0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.913023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.030466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3c3135-20d3-4c97-b88f-8b951d0480b0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.031867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3c3135-20d3-4c97-b88f-8b951d0480b0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.050863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.075113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c958cc-9a11-474d-b8a0-7199d5c8a4f2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.110779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.112543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.113027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.113445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.125820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.127289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-cf986edd-2f48-4ff1-8b74-162b0f757f22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:46.141573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:46.142695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:46.143588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:46.143588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:46.143588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:46.143588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 305/624] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:46.143821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 320/625] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:46.144845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4905534-ad1c-427b-b90e-ba06642701d7 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.145163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4905534-ad1c-427b-b90e-ba06642701d7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.145436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4905534-ad1c-427b-b90e-ba06642701d7 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.146519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ea5d55-2e27-4c7a-8738-249b3afad76d None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.146875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 321/626] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:46.147376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ea5d55-2e27-4c7a-8738-249b3afad76d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.147658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ea5d55-2e27-4c7a-8738-249b3afad76d None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.147832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82330ed-201e-4a0c-a4e2-786411af28f7 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.148148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82330ed-201e-4a0c-a4e2-786411af28f7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.148426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82330ed-201e-4a0c-a4e2-786411af28f7 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.149085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 306/627] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:46.150511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ea5d55-2e27-4c7a-8738-249b3afad76d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.154098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 322/628] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:46.155688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82330ed-201e-4a0c-a4e2-786411af28f7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.216303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.251256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ea5d55-2e27-4c7a-8738-249b3afad76d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.254564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82330ed-201e-4a0c-a4e2-786411af28f7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.267067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.283865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.284402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.284678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.284982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.285609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:46.285911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.287262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d281c8-1452-4b5d-907f-615936e93cf2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.289398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.290077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.290355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.290626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.298844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 307/629] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:46.300064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca7be1-3f63-4deb-be30-ceebd3d6bbdf None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.300275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.300389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca7be1-3f63-4deb-be30-ceebd3d6bbdf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.300611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca7be1-3f63-4deb-be30-ceebd3d6bbdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.301737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-74df2ac3-a0de-41ae-97bf-93f2f9b0f7a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:46.317120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:46.318262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:46.319352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:46.319352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:46.319352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:46.319352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 323/630] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:46.319352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.319706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.319848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.320975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-13ca7be1-3f63-4deb-be30-ceebd3d6bbdf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:46.322310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-13ca7be1-3f63-4deb-be30-ceebd3d6bbdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:46.337819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:46.338993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:46.388965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 308/631] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:46.390205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.390487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.390760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.410676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 324/632] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:46.411778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.419390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:46.420640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:46.480894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.504907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 309/633] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:46.504907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.530007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.540941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7733c169-8c2b-4f7f-9afc-d33b115d95e4 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.610728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.611360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.611634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.611901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.622320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.623774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:46.642564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:46.643490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:46.644962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 325/634] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:46.649341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.661925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64fc674-f32f-494d-8166-7bd9d0ec6aaa None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.705858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.706516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.707303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.707602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.708245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:46.708595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.710449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.725030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 310/635] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:46.726147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.773675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f1fda-35c3-4c3a-aacf-76c5ced55e24 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.780431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.810383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.810908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.811186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.811457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.812154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:46.812457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:46.814029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:46.825623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 326/636] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:46.826423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.837991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1518c63f-9fe1-46fd-826d-1483686b8e92 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.841566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.872566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.873220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.873502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.873776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.883594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:46.888802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:46.889790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:46.890609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 311/637] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:46.891301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.926740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab14e845-c36e-47d0-871d-0a504d540635 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.963495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:46.964111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.964382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.964645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.974439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:46.979528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:46.980495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:46.981345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 327/638] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:46.982021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:46.989264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b7035-bd76-4325-a8f1-b2a3aea0a73c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:46.996607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.026477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.027093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.027366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.027633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.034211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.035633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:47.047414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:47.048297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:47.049228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 312/639] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:47.049710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.058022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.084805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91dd2a2-5b5e-4dfe-be89-da8de029c567 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.118257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.118876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.119153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.119424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.129246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:47.134400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:47.135415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 328/640] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:47.136254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.157647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3478b-2ba5-4144-bf2f-6331cd1957cc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.193618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.194278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.194556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.194831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/10649368308f4dc4b8b887319cbcac8f/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.209102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=10649368308f4dc4b8b887319cbcac8f, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.210524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.223549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 313/641] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:47 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/10649368308f4dc4b8b887319cbcac8f/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:47.224455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.241121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9657d9-08b7-4e9b-8fe7-9a6bc7ae8ff3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.245513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.283170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.283803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.284088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.284363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.285091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.285397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.286842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.288461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.297883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 329/642] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.298755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.348320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd71ab-5455-4a5a-8dbe-d9f778fb91c5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.388264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.388898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.389187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.389461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.390160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.390473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.391840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.405263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 314/643] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.406119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.414793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c43f54a-c506-429e-91e8-3c61d9540735 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.452370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.453116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.453391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.453656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.454344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.454666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.456157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.468619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 330/644] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.473797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.495969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.519231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbe8911-e13d-4504-949d-dc1558ae16b8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.533189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.559309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.559913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.560191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.560461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.561146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.561446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.563090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.575746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 315/645] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.576794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.583452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9808c4-c1f2-4304-9538-f18964380fd3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.619825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.620548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.620960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.621334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.631912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.633439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:47.647608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:47.648917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:47.650727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:47.650727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:47.650727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:47.650727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 331/646] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:47.651371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.701021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29acbd03-e009-4cd8-a105-a30cecadfe1d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.736827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.737837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.738135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.738396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.748187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.750466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.752280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.762376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d737e2b-3e31-4a94-9572-e16d474b05ed None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:47.770223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:47.771362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:47.772412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:47.772412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:47.772412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:47.772916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.778894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 316/647] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:47.778894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.798509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.799119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.799397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.799666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.800378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=e528956ee4704ac0b5ce4c62826b71ee) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.800683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.802292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.809276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 332/648] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/role_assignments?user.id=e528956ee4704ac0b5ce4c62826b71ee&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.810056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.926515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4578307-e859-4a54-951e-9fbcaee4df73 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.928056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffbf6c3-2fa7-4b5f-b783-ba9a0d6ee547 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.973283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.974736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.974902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.975279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.976378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.976927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.979950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.989993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.990681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.990963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.991207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.004908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 333/649] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.005130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.005284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.005833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:48.030444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:48.031625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:48.032515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:48.032515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:48.032515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:48.032515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 317/650] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:48.032515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.057032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.091875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.152764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdf03d4-ef63-4b99-b28a-acd02ba7fbb3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.167413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63460579-1f4f-46e3-a975-ba2312a2a48b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.188685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.189264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.189539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.189804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.190440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:48.190741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.192481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.205688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 334/651] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.206687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.208152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.208850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.209253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.209520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.218042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.219488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:48.232178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:48.233382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 318/652] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/users/neutron => generated 85 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:48.234494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.312064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.322440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099283f4-c003-4c7f-8b6b-a37182ccde43 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.355829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6a415c-36db-4310-bdb4-4d80b18b02fc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.358494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.359476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.359476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.359934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.360674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:48.361104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.362580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.365221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.381278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 335/653] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.382279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.395731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.396417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.396734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.397068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] PATH_INFO: `/v3/users/glance` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.407374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.409164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:48.422967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:48.424107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 319/654] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/users/glance => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:48.425124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.503918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5f3e5-b633-420b-b4ae-170a0465dcfd None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.539606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a715ed-236b-43fc-b6bb-d440d1f1d88e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.546299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.547106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.547960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.548403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.549314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:48.550121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.552177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.580141 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 336/655] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.581292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.586945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.589959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.589959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.590476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.595608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.603096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.605057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:48.619143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:48.620080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:48.621035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 320/656] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/users/cinder => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:48.621646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.652729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.705499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af790b-5e11-4aea-9002-1b07515a7e67 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.740003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e49caf3-3108-49db-8120-25714ce40d7f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.749255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.750082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.750504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.751282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.752162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:48.752943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.754843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.777978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 337/657] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.779225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.780462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.781230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.781685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.782442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.857839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.859446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.867179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.889607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.913829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6bd419-0603-4ae7-8ef0-01f9c2bb735c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.954903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.955941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.956413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.956899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.957933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:48.958446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.960191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.979648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 338/658] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.980848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.998142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 321/659] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:48.998142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.095842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b385e6e3-de71-4d1e-b38f-699d7a680ec9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.111649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b5bb25-c36e-4603-b112-70daa34bb13e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.123656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.135046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.136041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.136521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.137150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.137803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.150966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.151575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.151855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.152126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.215051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.217317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.228700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.230214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.336198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 339/660] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:49.337655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.353935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 322/661] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:49.355044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.375760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.385849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.460394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f3c29-e997-45ad-98be-c5fc1b16b206 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.470746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe3ae74-ec62-48b3-9cb2-fe7140c03167 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.495078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.495727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.495999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.496266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.505819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.506130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.506438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.506718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.506987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.507561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.518314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.519747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:49.522510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:49.523435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:49.525764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:49.525764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:49.525764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:49.525764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 340/662] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:49.525764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-3e37c044-de7b-4796-990c-46ec83a53861 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:49.535290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:49.536215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:49.537104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:49.537104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:49.537104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:49.537104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 323/663] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:49.537843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee7ee6d-00ee-4daa-86f8-c6671defb5fe None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.538127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee7ee6d-00ee-4daa-86f8-c6671defb5fe None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.538402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee7ee6d-00ee-4daa-86f8-c6671defb5fe None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.543889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 324/664] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:49.544654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee7ee6d-00ee-4daa-86f8-c6671defb5fe None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.619359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.636213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.643622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceb76ce-da4f-40e9-a54d-7956d29c6b0c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.662809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee7ee6d-00ee-4daa-86f8-c6671defb5fe None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.677451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.678068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.678344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.678612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.690001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.691729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.699798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.700575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.701030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.701464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.702239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:49.702699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.704180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae099e3e-927f-4cc7-92f0-f70697417ab9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:49.706258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:49.707407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:49.708664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:49.708664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:49.708664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:49.711940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 341/665] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:49.713309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.720766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 325/666] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:49.721844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.722129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.722400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.742570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:49.743799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:49.821855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 326/667] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:49.821855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.835733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d104a33-8531-4995-bc3a-fda951e5067f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.874269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.887921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.888882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.889304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.889782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.891409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:49.891858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.893868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.908049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 342/668] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:49.909045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.911696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.957464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5713523-eed8-4431-93bf-e58184e44739 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.000152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.000799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.001113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.001652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.002304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.002610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.005155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.022272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 327/669] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.023810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.041726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f3edd-b8e2-4d78-a6bd-895a9539c420 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.088588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.089795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.090089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.090626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.100191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.101899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:50.118918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:50.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:50.120811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 343/670] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:50.121524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.159498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36436b2-1463-40a4-b3a8-67a44a198df5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.186202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.190051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.201072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.201704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.201981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.202252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.209089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.210529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:50.221990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:50.222924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:50.223923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 328/671] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:50.224531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.244364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060730c2-bee7-4f9f-9df7-211225fed48d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.276376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.277037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.277311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.277578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.283928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.285385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:50.297242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:50.298295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:50.299358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:50.302982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 344/672] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:50.303980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.326244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170d66e9-8cbe-4b84-a7e4-9fd9730e27b6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.359163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efadf3b3-f67a-4e10-bcfd-db928a02e843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.359775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efadf3b3-f67a-4e10-bcfd-db928a02e843 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.360048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efadf3b3-f67a-4e10-bcfd-db928a02e843 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.360316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efadf3b3-f67a-4e10-bcfd-db928a02e843 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.369921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efadf3b3-f67a-4e10-bcfd-db928a02e843 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-efadf3b3-f67a-4e10-bcfd-db928a02e843 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:50.374978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:50.375893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:50.376725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 329/673] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:50.377492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e41422c-ac89-4aaa-962d-b5f6607e8e58 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.377492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e41422c-ac89-4aaa-962d-b5f6607e8e58 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.377961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e41422c-ac89-4aaa-962d-b5f6607e8e58 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.379022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 330/674] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:50 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:50.382540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e41422c-ac89-4aaa-962d-b5f6607e8e58 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.401107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507064e7-6a53-4efe-af4c-ddfa3c673df6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.411247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.430002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.435444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.436050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.436333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.436606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.437334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.437640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.439103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.450815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 345/675] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.451591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.484989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e41422c-ac89-4aaa-962d-b5f6607e8e58 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.518564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.519289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.519683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.519955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.521537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.521863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.523293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.535133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 331/676] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.535914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.556367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63bc35-5f0f-4deb-94c2-1896d6d3d57e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.594314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.594921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.595200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.595472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.596157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.596461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.597922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139dc113-752a-475d-be2a-2bd1567e05f9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.609751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 346/677] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.610732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.611015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.611287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.629883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:50.631076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:50.639533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f67995-ccd5-47a2-9f97-127c93af95fe None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.646605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.654591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.673409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.673948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.674229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.674504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.675160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.675462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.676822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.685031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 332/678] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.685031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.701733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 347/679] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:50.702638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.804970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa777a-103f-4724-a591-91f7389d7ec5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.815719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e77f42-63a7-4ad8-8b1d-815bf4c177a0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.844747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.845404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.845679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.845963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.846658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=afdc21c25f734a51aa520e7d4e53455b) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.846961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.848367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.854688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 333/680] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/role_assignments?user.id=afdc21c25f734a51aa520e7d4e53455b&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.855457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.859799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.860832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.860832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.861043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.861668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=0298fb95ae9341c6a1eb8da546d93cb2) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.861971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.863499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.877405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 348/681] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/role_assignments?user.id=0298fb95ae9341c6a1eb8da546d93cb2&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.878569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.894502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.936515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.974289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2695b224-441b-4fde-95c4-060ffb856f1d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.016065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.016690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.016996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.017271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.018021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=09c034ad1ccd45e78a4b3aeab2129227) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.018321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.019831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.033053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 334/682] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/role_assignments?user.id=09c034ad1ccd45e78a4b3aeab2129227&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:51.034107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.055232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b2bba7-474b-4ff7-b7d2-863be4ad6911 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.107342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.108079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.108367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.108704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.124237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.126417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:51.144972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:51.146076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:51.146983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:51.146983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:51.146983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:51.146983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 349/683] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:51.155092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.216257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.224574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebf59a5-e7f8-41e9-b874-6a0630776b7b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.271768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.272939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.273429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.273962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.285760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:51.291389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:51.292334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:51.293411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 335/684] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:51.300708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.302698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.330324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945dc25-33a9-48b8-b1eb-3c911b810431 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.375975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.376578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.377020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.377302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.378442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.378747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.380401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.395631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 350/685] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:51.402031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.422194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1074272e-d7b1-4876-af2c-55c9a5ec95c6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.463063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.463804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.464087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.464360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.465232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.465540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.466982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.476289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 336/686] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:51.484220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.537263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.577796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a7e77c-c65a-4944-bda3-1ce7b4273414 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.647950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c49e738-c23e-4d0b-a6da-7e478c801d71 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.669578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.670221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.670498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.670769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.681333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.682921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.684371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.697090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.697719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.697994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.698263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:51.705156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:51.706078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:51.707400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 351/687] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:51.713146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.718878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.720253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:51.744251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:51.745290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:51.747428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:51.747428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:51.747428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:51.747428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 337/688] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:51.751367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.841288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.880684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6febc8-b5b6-4271-a428-c4635c3e0297 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.909686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b2898c-e2bb-4181-9593-744e05eb2a35 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.945689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.946327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.946601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.946869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.947588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.947886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.949339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.961584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.962243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.962520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.962786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.963427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.963725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.965126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.967925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 352/689] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:51.973605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.986767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 338/690] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:51.994279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.036132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.086215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e0fb6-d841-4049-ab92-84e3a9bc7383 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.121076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.121765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.122038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.122309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.194482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.196030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.214796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da807e0-8f7e-42bb-88de-7bb0fc470e82 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.265693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.279034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.279665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.279948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.280223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] PATH_INFO: `/v3/users/nova` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.281984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.296100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.298178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:52.321310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:52.322279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:52.323782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 339/691] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:52.328944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.362295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 353/692] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:52.371623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.481235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd017b49-8b0f-47c0-903e-8117395b86df None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.516996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229c7f7-f43a-41e6-bd66-06654a176cb1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.533597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.534217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.534494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.534763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.535486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:52.535788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.537225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.556112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 340/693] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:52.563578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.564674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.564995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.565275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.565843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.578383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.580155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:52.599025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:52.599980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:52.601503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:52.601503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:52.601503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:52.601503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 354/694] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:52.606176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.625260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.637710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.684924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d572a173-1439-49f8-a004-51e509fa0e13 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.755323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.756197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.756476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.758025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.797024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a0bf8b-4de2-4532-ab2a-6b76cacf310a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.858928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.859532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.859807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.860078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.860710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:52.861033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.865284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.884023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.886269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.889783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 355/695] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:52.895260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.982310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.999817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.042772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af805e2d-f761-437e-b619-2dd407bbe7b0 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.090130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 341/696] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:53.091988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.092641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.097140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.097140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.097359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.109607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.111272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:53.131786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:53.132944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:53.134026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:53.143627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 356/697] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:53.147077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.298570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d2e69f-629d-49b1-82f9-a531c66b2c90 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.333936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53fa2c-b2cb-4a4e-8919-13679674e28d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.370673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.371226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.371505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.371765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.385065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.394937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.396390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.401191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.401795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.402070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.404997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.405710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:53.406019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.407434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.419641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:53.427288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:53.428228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:53.429835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:53.429835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:53.429835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:53.429835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 342/698] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:53.436263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.436920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 357/699] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:53.442114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.576253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1b12a5-bd7c-4ee9-9d97-6178ea06af9b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.610584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4186b654-3683-4d3a-938a-a667ba9d8dfd None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.627980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.628530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.628814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.629233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.630005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:53.630317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.631730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a4d2ed-fa35-4bd7-a3fd-61c01123836a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.652894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 343/700] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.652894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a805266-91ac-4893-920a-9e3481247cb4 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.652894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a805266-91ac-4893-920a-9e3481247cb4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.652894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a805266-91ac-4893-920a-9e3481247cb4 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.652894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 344/701] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:53.652894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a805266-91ac-4893-920a-9e3481247cb4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.658407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.659218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.659636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.660039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.661036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=10649368308f4dc4b8b887319cbcac8f) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:53.661480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.663178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aa47f3-b006-4309-904f-1b6cdf72893e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.677397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 358/702] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/role_assignments?user.id=10649368308f4dc4b8b887319cbcac8f&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.679155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7bfe26-eae7-447b-ad2c-b5ed0e413f3a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.679582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7bfe26-eae7-447b-ad2c-b5ed0e413f3a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.680141 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7bfe26-eae7-447b-ad2c-b5ed0e413f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.711663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ff7bfe26-eae7-447b-ad2c-b5ed0e413f3a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:53.713740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ff7bfe26-eae7-447b-ad2c-b5ed0e413f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:53.723178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.770579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a805266-91ac-4893-920a-9e3481247cb4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.776020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.804908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 359/703] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:53.804908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a62f314-c501-405f-a824-af6ce294f01c None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.804908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a62f314-c501-405f-a824-af6ce294f01c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.804908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a62f314-c501-405f-a824-af6ce294f01c None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.804908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 360/704] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:53.804908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca60c35-957b-419e-9823-d417273cc54d None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.805414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca60c35-957b-419e-9823-d417273cc54d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.805414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca60c35-957b-419e-9823-d417273cc54d None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.811818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 361/705] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:53.811818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.811818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.811818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.820319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.821127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.821513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.821858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.830003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.830692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:53.831731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.831920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-e750b7f6-c68e-4167-bfaf-b01bcc79babe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:53.845339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:53.846424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:53.847356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 345/706] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:53.848579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.848889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.849170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.870062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:53.871749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:53.919610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 362/707] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:53.920975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.948944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 346/708] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:54.027310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9f59e2-a363-4719-834d-3b41de56016e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.038926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.063738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.064699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.065195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.065666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.066800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.067320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.069001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.081858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 363/709] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.087512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.110576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.146469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.211869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa82136-fa66-4cc2-990c-690fd3b04fe2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.221291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f91021b-15b8-44eb-aed7-a833c2917f39 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.255129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.255732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.256009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.256409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.257258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=e528956ee4704ac0b5ce4c62826b71ee) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.257568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.259593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.262489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.263117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.263395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.263667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] PATH_INFO: `/v3/services/network` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.264243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.265284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 347/710] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/role_assignments?user.id=e528956ee4704ac0b5ce4c62826b71ee&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.265623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.266079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:54.285408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:54.286512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:54.287767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 364/711] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/services/network => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:54.288575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.333284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.400565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c136d998-b9e2-4599-8cd0-0bb7bbd61355 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.417824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d6c032-6a3e-4436-828c-47af36b5ef99 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.418820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.447243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.448029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.448272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.448550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.449166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.450749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.462595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.463548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.464016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:54.464339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:54.465403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:54.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 348/712] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:54.472049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.472125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.484115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.485670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:54.506551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:54.507593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:54.508488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:54.508488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:54.508488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:54.512944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 365/713] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:54.513823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.596689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c5ef8-3f78-4e0e-a737-5223e3096d61 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.619360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.639627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfb895-2d01-4e3a-9f73-3929123c1662 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.641541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.642221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.642507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.642801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.644349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] RBAC: Adding query filter params (name=network) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.644660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.646080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.653207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 349/714] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/services?name=network => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.654058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.681069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.681685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.681963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.682237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.682863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.683174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.685086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.692270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 366/715] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.693246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.709001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.786668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f853a-be6c-45b8-af05-2a589947a4c5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.815982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e0dfc9-00f0-4b4c-b062-64523f7977c6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.830385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.831000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.831283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.831559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.832215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.832513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.833913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.852168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 350/716] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.852168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.869697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.871029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.871457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.872137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.872962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] RBAC: Adding query filter params (type=network) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.873675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.875696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.884899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 367/717] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/services?type=network => generated 124 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.886199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.907975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:55.019344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76913d39-29c3-4cc8-b6a6-3394eff553a8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:55.029430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:55.063707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c9432-d2bc-4689-a0fb-fe3996f025c3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:55.102868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:55.103285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:55.103567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:55.103840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:55.107513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:55.107809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:55.109226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0f46ab-ec7c-4cb6-afc2-96e33f315ce0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:55.123099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 351/718] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:55.130977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13570d63-c57d-46d0-bce0-03e29233b6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:55.131616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13570d63-c57d-46d0-bce0-03e29233b6f0 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:55.131896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13570d63-c57d-46d0-bce0-03e29233b6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:55.132171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13570d63-c57d-46d0-bce0-03e29233b6f0 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:55.132841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13570d63-c57d-46d0-bce0-03e29233b6f0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:55.134375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13570d63-c57d-46d0-bce0-03e29233b6f0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:55.170640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 368/719] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:54 2022] POST /identity/v3/users => generated 254 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:56.442042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2079b782-abe4-40ce-be7c-bddd0321749e None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2079b782-abe4-40ce-be7c-bddd0321749e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2079b782-abe4-40ce-be7c-bddd0321749e None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.444150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 352/720] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:56.449613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b6f25a-2541-4a8d-822e-6ddb264b673d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.450003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b6f25a-2541-4a8d-822e-6ddb264b673d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.450183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b6f25a-2541-4a8d-822e-6ddb264b673d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.478396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-49b6f25a-2541-4a8d-822e-6ddb264b673d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:56.479583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-49b6f25a-2541-4a8d-822e-6ddb264b673d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:56.555408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:56.565930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 369/721] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:56.846197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610dee12-a2b7-4e35-be8a-c1ff020737cd None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.846612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610dee12-a2b7-4e35-be8a-c1ff020737cd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.846793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610dee12-a2b7-4e35-be8a-c1ff020737cd None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.848286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 353/722] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:56.853886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.854179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.854460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.881871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:56.883076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:56.912201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610dee12-a2b7-4e35-be8a-c1ff020737cd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:56.985261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 370/723] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:57.073228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610dee12-a2b7-4e35-be8a-c1ff020737cd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.121970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.122529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.122818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.123093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.140084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:57.145281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:57.146200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:57.147617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 354/724] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:57.152187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.285013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c92c7-c40c-4cd7-ac20-a2ba47e618f5 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.322402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.335545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.336184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.336470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.336775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.337587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:57.338145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.340680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.351067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 371/725] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.357333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.382844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.406841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.505075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c53732a-57c5-4df5-aabd-7d1d1249e513 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.527137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cac136-f23b-429c-b939-d206bca72075 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.548530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615605cf-5ecc-4d15-9379-1dad6b57fdeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.549742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615605cf-5ecc-4d15-9379-1dad6b57fdeb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.550021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615605cf-5ecc-4d15-9379-1dad6b57fdeb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.550413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615605cf-5ecc-4d15-9379-1dad6b57fdeb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.564176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615605cf-5ecc-4d15-9379-1dad6b57fdeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:57.565271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.566392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.566870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.567336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-615605cf-5ecc-4d15-9379-1dad6b57fdeb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:57.569954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:57.570883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:57.572191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 372/726] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:57.572908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c4cd65-b0aa-4ee4-9864-35ef9e1466f8 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.572908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c4cd65-b0aa-4ee4-9864-35ef9e1466f8 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.577244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c4cd65-b0aa-4ee4-9864-35ef9e1466f8 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.588806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 373/727] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:57 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:57.588806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c4cd65-b0aa-4ee4-9864-35ef9e1466f8 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.593073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.594771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-701acffc-9120-4779-bf13-098894e96cfb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:57.610468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:57.611421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:57.612336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:57.612336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:57.612336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:57.614532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 355/728] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:57.615995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.616498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.617011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.637441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:57.638960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:57.657260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.718386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 356/729] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:57.718386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.722295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c4cd65-b0aa-4ee4-9864-35ef9e1466f8 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.740938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.794035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.795119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.795626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.796142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.797055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:57.797564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.799231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.819395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 374/730] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.825716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.871137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9301702-197a-4607-8b39-2643070cfa91 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.911775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.912304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.912579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.912845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.913512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:57.914228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.916422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e7fdc5-4d9f-422d-af78-71c5015e2386 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.931278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 357/731] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.932682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc2bc4-043c-4de2-85f8-59ae60c61b9f None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.933018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc2bc4-043c-4de2-85f8-59ae60c61b9f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.934066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc2bc4-043c-4de2-85f8-59ae60c61b9f None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.935338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 358/732] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:57.936127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc2bc4-043c-4de2-85f8-59ae60c61b9f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.965815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ee875-3c5c-414b-b91d-56f412680e0f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.981291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.004765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.005847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.006362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.006902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.019699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.021629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.028299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-9891a465-cd7b-48c4-8a52-2b0214821b36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:58.039700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:58.041389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:58.042279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:58.042279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:58.042279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:58.044608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 375/733] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:58.046093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.046575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.047059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.065716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:58.067195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:58.073879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc2bc4-043c-4de2-85f8-59ae60c61b9f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.108891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.109520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.109800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.110070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.117743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.119995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:58.133456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:58.134531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 359/734] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:58.135968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.142633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 376/735] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:58.143461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.237428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.241030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8818f4-f3ee-4a5c-80ba-b53a8a08fe46 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.244435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0127856-904c-4dd7-9db3-d97322e4bfdc None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.267431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.275929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.276526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.276810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.277103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.277728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:58.278025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.279364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2528dd8-336e-45d4-ba27-728f488cce00 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.281953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.282555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.282836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.283104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.283746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.285134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.295651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 360/736] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:58.296960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e190e31-86dc-4f99-82e9-04613bec02b6 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.297249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e190e31-86dc-4f99-82e9-04613bec02b6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.297538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e190e31-86dc-4f99-82e9-04613bec02b6 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.298753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 361/737] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:58:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:58.299434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e190e31-86dc-4f99-82e9-04613bec02b6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.301907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 377/738] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:58:58 2022] POST /identity/v3/services => generated 240 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:58.303231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.433204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e8c150-467f-474a-af2f-a80458e14dd8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.453948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e190e31-86dc-4f99-82e9-04613bec02b6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.471254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.471898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.472177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.472449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.480303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.481775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-33bfd46f-c0d3-4c13-8d06-16bff9fbf294 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:58:58.493820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:58.494849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:58.495796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 378/739] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:58.496421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.496577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.496829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.500743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.501545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.501821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.502156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.502871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:58.503176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.504650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.515944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:58.517176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:58.523623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 362/740] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:58.524607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.525128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.585842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.604915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 379/741] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:58 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:58.604915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.658604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf456dc-0ab8-489e-86ce-c97f3a1b452e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.699107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.700775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.701195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.701550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.702285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.703732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.725407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 363/742] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:58:58 2022] POST /identity/v3/services => generated 251 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:58.726366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.748701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd65924-7a53-48e4-bc38-e1ba96be1ce0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.799392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.800016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.800298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.800638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.801572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:58.801881 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.803575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.814394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.828674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 380/743] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:58.837319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.958361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254c8aa-53ea-487f-8d65-8f4a930e059b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.962418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998fa72-26b9-4364-83fe-85b9336801ee None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.971090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.002364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.002867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.003151 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.003426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.010132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.010835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.011733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.011733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.026173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.028101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:59.045816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:59.046827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:59.047721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:59.047721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:59.047721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:59.049984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 381/744] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:59.051000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.088800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.090292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.111940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.169586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f143e9ea-9972-4a35-9697-f5ca0dbe11d8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.197022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 364/745] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:58:59.199916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.207997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.208591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.209655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.209836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.210190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.227340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:59.233380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:59.234311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:58:59.235147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:59.238020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 382/746] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:59.239002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.310427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c0073c-2e3a-45fb-b723-2e5ae50907f6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.345757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.346361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.346643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.346919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.347560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:59.347865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.349399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.352963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ebde03-8ca6-4523-8b13-16361a14b502 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.365690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 365/747] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:59.365690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.367482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.396557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.397681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.398246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.398793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.410721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.412581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:59.428833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:59.429815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:59.430788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:59.430788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:59.430788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:59.431702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.434128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 383/748] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:59.435269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.470755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46abc233-22a8-43df-ab44-ba24af9151ea None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.514261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.514589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.514761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.515033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.515691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:59.515992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.517369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.526545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 366/749] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:59.527405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.549931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1520f3-2516-4ffc-afca-6057fb389fec None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.589903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.591496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.591977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.592445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.600240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.601960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:58:59.615318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:59.616236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:58:59.617149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:59.624213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 384/750] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:59.625578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.628615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b5f11-e76b-4975-924d-af1134cd6da3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.655195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.661494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.664714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.665605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.665888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.666155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.666798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:59.667097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.669297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.686810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 367/751] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:59.687682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.748623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea674e0-c306-4a6e-8e78-97d8dcdd14b8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.785423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.786464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.786940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.787406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.792295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f80589-177e-493b-8dc7-a725dfc071eb None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.798874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.800715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:59.818363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:59.819286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:59.820171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:59.820171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:59.820171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:58:59.822451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 385/752] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:59.823493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.826837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.828108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.828108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.828108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.828834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:59.829245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.830761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.847840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 368/753] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:59.847840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.892824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.913299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.940536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cebc531-01d6-4fba-86ea-35ed80c05d2d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.946877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dc452d-d1f7-4c4e-a9a2-d4f77b966fed None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.979952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.980894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.981445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.981891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.981996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.982787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.983195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.983474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.984111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:59.984408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.985799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.990548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.992216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.999643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 369/754] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:00.000553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:00.005397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:00.006255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:00.007213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:00.010175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 386/755] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:00.011161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.107454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2de5581-c17e-4e95-94d6-1dcfe1877e40 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.117042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.121216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62d497-20b6-4dec-a39d-742c524f949b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.147029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.147749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.148023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.148294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/e528956ee4704ac0b5ce4c62826b71ee/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.150361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=e528956ee4704ac0b5ce4c62826b71ee, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:00.151836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:00.155485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.162280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.163294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.163773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.164272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.172897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 370/756] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:59 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/e528956ee4704ac0b5ce4c62826b71ee/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:00.172897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.180428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:00.182331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:00.206879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:00.207872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:00.208803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:00.211409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 387/757] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:00.217270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.282838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fe2027-dc7d-4adc-b8c8-403c7bad0f1e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.320894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.320894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.320894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.321849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.322383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:00.322687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:00.324663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:00.338700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 371/758] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:00.344967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.400602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a927cda-31f1-4d7b-b5c2-f4508024d4cc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.434841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.442646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.443322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.443647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.443911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.444796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:00.445010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:00.446987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:00.470688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 388/759] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:00.476275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.529499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.546577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce132b73-f807-49a0-ace7-98f1db5e596d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.584381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54152636-ba4b-4294-89ba-10d9ce905942 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.592891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.592891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.592891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.592891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.593414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=10649368308f4dc4b8b887319cbcac8f) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:00.593697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:00.595695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6586a7e9-c767-42fe-9374-f5f82389c4b1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:00.604912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 372/760] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/role_assignments?user.id=10649368308f4dc4b8b887319cbcac8f&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:00.604912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff48f41-70e1-46c3-94f1-4c77b800f6e3 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.604912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff48f41-70e1-46c3-94f1-4c77b800f6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.604912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff48f41-70e1-46c3-94f1-4c77b800f6e3 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.605911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 373/761] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:00.612096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.612096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.612096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.631979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:00.633239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:00.640697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.642549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.642549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.647258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.755619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 374/762] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:00.783208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:00.785131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:00.863121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.919334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 389/763] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:00.924755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.951142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.038208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da73425a-97e0-4f1b-8de0-e71692825024 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.084905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.085743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.086094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.086429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.092517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.107262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.108804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-bd3e0afe-3cdc-4360-b935-8250aea048c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:01.127610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:01.128765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:01.129712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:01.129712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:01.129712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:01.133239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 375/764] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:01.134552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880f888-dae3-4af8-93a1-f1fd19ac7a02 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.134839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880f888-dae3-4af8-93a1-f1fd19ac7a02 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.135208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e880f888-dae3-4af8-93a1-f1fd19ac7a02 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.138017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 376/765] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:01.138742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e880f888-dae3-4af8-93a1-f1fd19ac7a02 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.220491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4493b1b6-e3f8-40e7-9380-12e97a09eb4b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.228364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.256633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e880f888-dae3-4af8-93a1-f1fd19ac7a02 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.267003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.267651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.268359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.268636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] PATH_INFO: `/v3/services/network` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.269543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.271190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-0d16f699-f6c7-4646-abfd-3a873f654229 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:01.284235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:01.285399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:59:01.286244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:01.287054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 390/766] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/services/network => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:01.292968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.292968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.292968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.295214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.295929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.296382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.296779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.297678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:01.298108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.299631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.311000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:01.312218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:01.313216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 377/767] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.314147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.330334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.402366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 391/768] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:01 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:01.403301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.470019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6e9083-957a-47be-b578-bc3dc6d7711e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.527483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.528113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.528426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.528734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.529421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] RBAC: Adding query filter params (name=network) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:01.529759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.531332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.535642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.541456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 378/769] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/services?name=network => generated 124 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.546961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.568388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff69ab0-a73d-407d-be78-0f6dc6e471c7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.635332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.636668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.638123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.638543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.656463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.658162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:01.670325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:01.671422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:01.672532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:01.674203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 392/770] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:01.681213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.687707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.704096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4a3cf3-98be-42b2-b0f1-3ff8fc780382 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.773535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.774441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.775046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.775550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.776459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] RBAC: Adding query filter params (type=network) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:01.777017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.778655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.785443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 379/771] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/services?type=network => generated 350 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.786436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.842073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1236580a-599b-4b7c-9fb6-1321f770d9a1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.866409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.888664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.889571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.889856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.890132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.891314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:01.891630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.893374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.911167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 393/772] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.912984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.921889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66360f55-123f-497e-b708-97abf0793f07 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.965203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.965607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.965898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.966223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.966816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.968193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:01.978560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:01.979536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:01.980373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 380/773] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:01.981185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.985880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.026815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6733dc1-3499-4911-9cff-0f5b58016e8b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.064081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.065066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.065583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.066067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.067281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] RBAC: Adding query filter params (interface=public, service_id=1a8e4890ec314833a25228addb9493d4, region_id=RegionOne) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.067787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.069380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.081926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 394/774] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/endpoints?service_id=1a8e4890ec314833a25228addb9493d4&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.083135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.097598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65dd7f9-a897-457a-97a3-51b2ffabb09e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.130936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.141871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.142858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.143378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.143846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.144763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=3546ff5fc9094d35921920f32e0c7476) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.145390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.147065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.155380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 381/775] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/role_assignments?user.id=3546ff5fc9094d35921920f32e0c7476&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.156625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.251396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c7ed1c-5bbe-4051-ba8c-9ccb73c2b6bd None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.313601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.329734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.333678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.334210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.334683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.335535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.336045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.337792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.348094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 395/776] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.353019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.367423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b479b22b-e0a6-4a12-ba2c-9e3ad57c3008 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.413748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.415335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.415625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.415902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.416813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.418233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.424691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 382/777] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/services => generated 772 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.484236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192e0b58-cec0-4581-b8d3-5bd342d9b623 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.521558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.522572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.523050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.523627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.524530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.525052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.526674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.532449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 396/778] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.540926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.637833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.695396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c0f22e-5999-4c9c-ae46-34d4ea688efe None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.733491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.734108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.734388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.734663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.735380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] RBAC: Adding query filter params (interface=public, service_id=4ff89729381742249a18f018a165bc16, region_id=RegionOne) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.735691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.737122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accad911-d1cd-44f2-a762-772448a9df2f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.748284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 383/779] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/endpoints?service_id=4ff89729381742249a18f018a165bc16&interface=public®ion_id=RegionOne => generated 194 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.753588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.833606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.849317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4364b77-1da4-4764-bba7-99539ae93e57 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.849616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4364b77-1da4-4764-bba7-99539ae93e57 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.849933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4364b77-1da4-4764-bba7-99539ae93e57 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.852632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 384/780] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:02.856706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04d964-0b67-4c20-9317-f8fc20f27fe6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.857011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04d964-0b67-4c20-9317-f8fc20f27fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.857284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04d964-0b67-4c20-9317-f8fc20f27fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.887144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ca04d964-0b67-4c20-9317-f8fc20f27fe6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:02.888409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ca04d964-0b67-4c20-9317-f8fc20f27fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:02.914284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcb25e0-b069-4a78-9f38-ad93cb3038ae None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.982379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae16a4f-dcdf-47ea-b285-b3dc8d4fd53e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.982995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae16a4f-dcdf-47ea-b285-b3dc8d4fd53e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.983276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae16a4f-dcdf-47ea-b285-b3dc8d4fd53e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.983551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae16a4f-dcdf-47ea-b285-b3dc8d4fd53e None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.984166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae16a4f-dcdf-47ea-b285-b3dc8d4fd53e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.985571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae16a4f-dcdf-47ea-b285-b3dc8d4fd53e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.000127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 397/781] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/services => generated 772 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:03.020995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 385/782] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:02 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:03.150799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aed042-7b23-4771-b093-5bab5cbc9665 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.151103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aed042-7b23-4771-b093-5bab5cbc9665 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.151282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aed042-7b23-4771-b093-5bab5cbc9665 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.153081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 398/783] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:03.158383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.158700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.158943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.182428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:03.183675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:03.233947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.262553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 386/784] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:03.363792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aed042-7b23-4771-b093-5bab5cbc9665 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.467996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aed042-7b23-4771-b093-5bab5cbc9665 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.488244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.503282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.503929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.504207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.504481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.515460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:03.520446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:03.521427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:03.522251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:03.524781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 399/785] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:03.528569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.633669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.634405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc921ba-6ead-4d22-94ce-993ac1a3dce8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.680651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.681711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.682218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.682663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.683496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:03.684033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.685839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.695979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 387/786] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:03.701863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.776419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.825684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da141330-9a0c-4c31-9395-eb50b1d1c655 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.873933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ac382d-c3b2-4703-a421-7c815f626614 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.900262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.900921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.901483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.901913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.922589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:03.931193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:03.932104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:03.932970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:03.938156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 400/787] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:03.941899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.942570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.942851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.943121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.945795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.970080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.971547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:04.002715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:04.003710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:04.004601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:04.004601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:04.004601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:04.006492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 388/788] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:04.011349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.035584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.187850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f77a87f-19ef-4968-a3ae-2e9654cae3db None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.258429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.259103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.259380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.259651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.260313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:04.260611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.262013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.279582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 401/789] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:04.286116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.361840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.407548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba33902f-d5cd-4201-a5f9-a963630dbb5f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.410496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288179db-1e62-4f64-9d9c-0d5668a47d1c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.413188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.447776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.448406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.448691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.449002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.460063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.461578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:04.477066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:04.478063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:04.478961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:04.478961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:04.478961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:04.478961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 402/790] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:04.484463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.560360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.560905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.561189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.561457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.562101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:04.562403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.573520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.603926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 389/791] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 593 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:04.614079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.635839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8910d444-30c5-4bff-aace-16164a668b79 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.678479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.679101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.679433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.679743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.681004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:04.681284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.682963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a608609-84df-4526-a31a-ad9abc2fea89 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.700934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 403/792] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:04.703071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0fea48-1342-4d08-a985-95471b5eebc4 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.703374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0fea48-1342-4d08-a985-95471b5eebc4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.703648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0fea48-1342-4d08-a985-95471b5eebc4 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.708945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 404/793] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:04.710912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0fea48-1342-4d08-a985-95471b5eebc4 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.733423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.743681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8926ae11-6de2-41f8-8974-66ed7e44f1f5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.779617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.781149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.781625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.782106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.790493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.792204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-62286d7f-047a-49c0-a1d8-2254e6d343f7 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:04.804156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:04.805153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:04.806156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:04.808603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 390/794] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:04.814588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d1736-36df-4d09-b748-8875b381805d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.815086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d1736-36df-4d09-b748-8875b381805d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.815564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d1736-36df-4d09-b748-8875b381805d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.835184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-816d1736-36df-4d09-b748-8875b381805d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:04.836758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-816d1736-36df-4d09-b748-8875b381805d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:04.841954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0fea48-1342-4d08-a985-95471b5eebc4 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.845739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.895711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.896345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.896625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.896915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.906827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.909163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.919107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 391/795] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:04 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:04.920320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d1736-36df-4d09-b748-8875b381805d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:04.927808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:04.928732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:04.930007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:04.937159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 405/796] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:04.939112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.079991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.110733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae9adb-6afb-42c3-86f4-db81873eb65a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.131620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d1736-36df-4d09-b748-8875b381805d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.151630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.152243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.152518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.152790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.153601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:05.153901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.155342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.174288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 406/797] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.179350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.209128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.209733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.210011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.210276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.210990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:05.211290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.213269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a46c3a-9e2d-48a9-8e82-ac6b46aa2df1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.231616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.231960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 392/798] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.236716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda09534-6222-4234-bd9c-253b106ee0ee None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.237056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda09534-6222-4234-bd9c-253b106ee0ee None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.237330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda09534-6222-4234-bd9c-253b106ee0ee None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.239031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 393/799] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:05.239765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda09534-6222-4234-bd9c-253b106ee0ee None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.302378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f12868-831f-4c86-9c0e-effd2e1a278f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.333367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.339178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.339797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.340069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.340334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.350748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.352189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.362908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda09534-6222-4234-bd9c-253b106ee0ee None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-bb0472c1-c68e-4b5a-ac25-f4f656733635 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:05.371075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:05.372099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:05.373015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:05.373015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:05.373015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:05.373015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 407/800] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:05.375118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.375403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.375669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.399118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:05.400303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:05.405903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.406497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.406771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.407033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.469989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 408/801] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:05.470843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.491686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.495984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.498726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.584328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4736bacd-3fa6-4ada-a166-e4296dd15bbd None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.600364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.622398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b75ea0-4198-4a88-9efa-e7277129c7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.623036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b75ea0-4198-4a88-9efa-e7277129c7f1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.623313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b75ea0-4198-4a88-9efa-e7277129c7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.623590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b75ea0-4198-4a88-9efa-e7277129c7f1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.633681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b75ea0-4198-4a88-9efa-e7277129c7f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:05.640468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 394/802] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:05.640468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b5b75ea0-4198-4a88-9efa-e7277129c7f1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:05.640692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:05.641590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 409/803] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:05.642431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc083660-1ba4-4546-b3b8-b5ab03e5e957 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.643168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc083660-1ba4-4546-b3b8-b5ab03e5e957 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.643168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc083660-1ba4-4546-b3b8-b5ab03e5e957 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.644994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 410/804] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:05.646933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc083660-1ba4-4546-b3b8-b5ab03e5e957 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.756847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.759763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa807b7-f203-40bc-bf72-3ddcb4525145 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.768405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc083660-1ba4-4546-b3b8-b5ab03e5e957 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.795253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.795852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.796127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.796397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.797070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:05.797369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.798773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.801576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.802173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.802445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.802710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.803354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:05.803647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.805015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f95f608-0a38-4ef0-a3aa-f84d4eb0ad78 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.809576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 395/805] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.810597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.820436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 411/806] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.820436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.820436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.820436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.834846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:05.835950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:05.864286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.902128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 412/807] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:05 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:05.903054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.920239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1cb117-75d0-47ac-aa22-74382af7750c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.957768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.958388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.958668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.958939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.969153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.970577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:05.984131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:05.985308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:05.986648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:05.986648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:05.986648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:05.989841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 396/808] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:05.990680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.004831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82e58b2-cbca-4226-84af-efbf98a7949f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.005827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.050832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914af089-6389-408f-8349-12bf23504736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.052288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914af089-6389-408f-8349-12bf23504736 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.052615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914af089-6389-408f-8349-12bf23504736 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.053370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914af089-6389-408f-8349-12bf23504736 None admin] PATH_INFO: `/v3/services/network` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.054053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914af089-6389-408f-8349-12bf23504736 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.055989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914af089-6389-408f-8349-12bf23504736 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-914af089-6389-408f-8349-12bf23504736 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:06.066510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:06.067478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.068320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 413/809] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/services/network => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:06.069146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914af089-6389-408f-8349-12bf23504736 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.098977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.100428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24c3c5-22bb-4faa-aa9e-388041d5e980 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.134777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.135396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.135838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.136145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.144702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.146216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:06.158853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:06.159845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:06.160785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:06.164769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 397/810] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:06.166146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.174331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914af089-6389-408f-8349-12bf23504736 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.210270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.210883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.211164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.211428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.222161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.223602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.235372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:06.237823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:06.238817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:06.239711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:06.239711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:06.239711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.239711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 414/811] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:06.239711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.268457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a6857-0129-4753-9ebd-92ae8c73fe34 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.300742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.301395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.301676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.301950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.302588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.302892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.304256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.316210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 398/812] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.317283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.322294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.341383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b8e91-897b-4c68-afd3-58369529799e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.377208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.377740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.378026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.378301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.378929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] RBAC: Adding query filter params (name=network) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.379229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.380594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.385305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 415/813] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/services?name=network => generated 124 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.386089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.419767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec289d4-3aae-4291-a86a-f3f3a6e94c92 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.448525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.453718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.454319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.454599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.454867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.455551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.455853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.457295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.469266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 399/814] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.470052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.483799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac0dcb-9fd2-4179-b75a-56bc722485a6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.516446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.517109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.517499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.517772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.518346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.519710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:06.528781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:06.529802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.531076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 416/815] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:06.532010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.543419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.572059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e45d124-8a21-4b7e-a52e-33fe2202bedc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.605147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.605748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.606028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.606298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.606935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.607236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.608600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.620681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 400/816] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.621497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.628931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87757bb9-6ab8-478d-98ee-2eb6b267c956 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.662431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.663050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.663324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.663589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.670389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.671945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.672329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:06.683596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:06.684658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:06.685788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 417/817] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:06.687236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.723304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde7e112-00d8-4aa2-b8ec-9675a97feae6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.755407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.761717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.762323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.762603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.762874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.763496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] RBAC: Adding query filter params (type=network) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.763795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.765172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.770002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 401/818] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/services?type=network => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.770685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.789348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1454d44-4047-4614-89e6-b18c1dcacc34 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.822434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.823078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.823352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.823620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/10649368308f4dc4b8b887319cbcac8f/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.825510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=10649368308f4dc4b8b887319cbcac8f, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.826927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.838955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 418/819] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:06 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/10649368308f4dc4b8b887319cbcac8f/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:06.845872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.879609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841e2e0e-a045-446e-bae0-c1830d460aa2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.903831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.927491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.928213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.928563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.928933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.929767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.930146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.931829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.937818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 402/820] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.938812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.965200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f93364-bd07-45b0-95bb-94c06b8394e6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.999395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.010674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.011765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.012044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.012314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.023163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.025257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:07.039345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:07.040552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:07.041727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 419/821] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:07.042423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.068621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585917e2-4c14-4efd-a0e1-163a94fe6ace None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.104958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.105567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.105913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.106116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.106798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:07.107104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.109070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.121194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 403/822] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:07.121837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.153968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4c5bb-6014-43a1-988d-475eeed5d864 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.165489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.193264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.194465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.194917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.195483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.197185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.198719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.226197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 420/823] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:07 2022] POST /identity/v3/endpoints => generated 337 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:07.227296 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.227934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfdb3da-a632-4c65-9838-605b70c2a50f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.295828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.300279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.301326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.301767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.302381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.303750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:07.304269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.305962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.312665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 404/824] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:07.313901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.381782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fbd014-89fe-4a04-a59b-eb1c3ab07ad9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.431106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.431914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.433040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.433040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.434583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.435338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9bbb10-dea1-4265-ad63-efdbf5dcd63c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.436919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:07.436919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.438092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.474935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.475669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.476027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.476368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.477278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=e528956ee4704ac0b5ce4c62826b71ee) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:07.477688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.479241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.486207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 405/825] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/role_assignments?user.id=e528956ee4704ac0b5ce4c62826b71ee&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:07.487157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.580630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 421/826] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:07.592897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.623527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031e4b2f-70bc-41aa-b1c0-94f027d50819 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.661461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.662388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.662388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.662659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.663286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.664674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.691809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 406/827] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:07 2022] POST /identity/v3/endpoints => generated 346 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:07.737837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81870ae4-494d-4253-a9c4-75081587606f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.741705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.790443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.791117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.791391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.791655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.803602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.805101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:07.822305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:07.823334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:07.824233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:07.824233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:07.824233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:07.824233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 422/828] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:07.830792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.856076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.942545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee3a1d5-ff18-47a6-bbab-b9d5ecbebe31 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.976585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.977251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.977564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.977829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.978465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:07.978764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.980114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.993522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 407/829] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:08.005387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:08.109484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4982a65c-27f0-425b-a364-0fdc85542afa None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:08.146041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:08.146681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.146956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.147229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.154745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:08.156198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:08.167799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:08.168702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:08.169655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 423/830] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:08.176470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:08.267386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:08.286492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe7cf80-9322-4986-97c2-819fc753f2ba None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:08.320558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:08.321709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.321987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.322253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.322928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:08.323226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:08.324631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3c42d-8baa-4865-a673-2197eb8c344a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:08.336862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 408/831] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:08.342713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:08.358852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:08.447488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6764c11-5962-4580-9a4f-61e12151500b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:08.481652 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0ec32e-b325-404c-bbb9-3d29a613440d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:08.482311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0ec32e-b325-404c-bbb9-3d29a613440d None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.482589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0ec32e-b325-404c-bbb9-3d29a613440d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.482859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0ec32e-b325-404c-bbb9-3d29a613440d None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/3546ff5fc9094d35921920f32e0c7476/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.497629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0ec32e-b325-404c-bbb9-3d29a613440d None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=3546ff5fc9094d35921920f32e0c7476, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:08.499079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0ec32e-b325-404c-bbb9-3d29a613440d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:08.513970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 424/832] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:08 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/3546ff5fc9094d35921920f32e0c7476/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:09.097862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e1ec0e-4ed1-4469-bcea-b5b8bd1486ac None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.098382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e1ec0e-4ed1-4469-bcea-b5b8bd1486ac None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.098382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e1ec0e-4ed1-4469-bcea-b5b8bd1486ac None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.099999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 409/833] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:09.105669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.106058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.106244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.123781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:09.124982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:09.180007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.192288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 425/834] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:09 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:09.459470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e1ec0e-4ed1-4469-bcea-b5b8bd1486ac None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.568129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e1ec0e-4ed1-4469-bcea-b5b8bd1486ac None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.602877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.603501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.603779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.604049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.611787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.615405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:09.621298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:09.622230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:09.623077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 410/835] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:09.628276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.726408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adcbff-0888-4acd-a68b-faa301ef7173 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.765282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.765890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.766167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.766439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.767101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:09.767398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.768780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869f5ca-7e67-4feb-8783-5b47a896c706 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.778758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 426/836] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:09.784467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.894205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0031cc9-cfc6-4f79-87c7-dade2d554511 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.903374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a2af0-2650-4262-a297-c3d3dcf83714 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.903658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a2af0-2650-4262-a297-c3d3dcf83714 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.904052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a2af0-2650-4262-a297-c3d3dcf83714 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.906870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 427/837] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:09.910911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759cfecd-ecbf-4c9f-af2a-95dabb8d3560 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.911291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759cfecd-ecbf-4c9f-af2a-95dabb8d3560 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.911570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759cfecd-ecbf-4c9f-af2a-95dabb8d3560 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.929261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.929883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.930159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.930487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.931699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-759cfecd-ecbf-4c9f-af2a-95dabb8d3560 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:09.933116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-759cfecd-ecbf-4c9f-af2a-95dabb8d3560 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:09.941247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.942938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-68acb116-2026-404c-87ee-4c13975cfd14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:09.957954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:09.958912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:09.959806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:09.959806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:09.959806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:09.959982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 411/838] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:09.965902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bba875-01d0-4c59-97c0-42f4d33204e5 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.965902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bba875-01d0-4c59-97c0-42f4d33204e5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.965902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bba875-01d0-4c59-97c0-42f4d33204e5 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.965902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 412/839] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:09.968053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bba875-01d0-4c59-97c0-42f4d33204e5 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.990347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.010198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 428/840] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:09 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:10.011405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.011683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.011950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.027805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.034877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:10.036056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:10.083494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bba875-01d0-4c59-97c0-42f4d33204e5 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.121070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.122411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.122911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.123389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.124211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:10.124706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.125427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 429/841] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:10.126853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.143180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 413/842] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.148750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.251328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae4af89-02f8-49e3-a405-b46649020e4e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.266145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.286988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.287615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.287888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.288153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.295194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.296625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:10.308699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:10.309694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.311046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 430/843] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:10.315946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.319056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.440793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d2509-4471-4728-ba08-a4da783b1cb3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.441869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d85721-2886-447c-9b14-da628d70ec13 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.454573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.478927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.479476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.479618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.479800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.480071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.480167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.480360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.480631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.480792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:10.481110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.481259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.482523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.482619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:10.494105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:10.495050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.495929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 431/844] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:10.496730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.503027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 414/845] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.504071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.544973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.606179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abac8384-171e-4a95-baac-bebd7e5dd4d3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.612518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004852f2-5c5a-4354-a6f7-78ddb9d92b8f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.643673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.644330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.644681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.644897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.645476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.646888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.652263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.652907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.653185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.653457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.654092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:10.654391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.655816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:10.656513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:10.657517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:10.658353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 432/846] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:10.659994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.672415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 415/847] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.672415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.706008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.766182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcf2776-d884-4243-a8d9-460bb3a5628a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.767581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.778468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd8b9f7-e804-478a-bd9b-896c5166a15a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.800137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38cbbd9-f6dd-412f-ab85-121608f1a981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.800740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38cbbd9-f6dd-412f-ab85-121608f1a981 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.801048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38cbbd9-f6dd-412f-ab85-121608f1a981 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.801320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38cbbd9-f6dd-412f-ab85-121608f1a981 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.810728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.811322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.811606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.811877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.812491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:10.812787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.814164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.818623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 416/848] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.819385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.868403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38cbbd9-f6dd-412f-ab85-121608f1a981 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.869878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38cbbd9-f6dd-412f-ab85-121608f1a981 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.915269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.919899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efae7bc-1a60-4f09-b7d1-db45e629ef4f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.953947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.954545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.954821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.955093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.955708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:10.956007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.957381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7468e65c-ade5-428a-9d91-790ce36fa374 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.967931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 417/849] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.967931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f933e8-0423-44e5-875f-467d7a02d62b None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.967931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f933e8-0423-44e5-875f-467d7a02d62b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.967931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f933e8-0423-44e5-875f-467d7a02d62b None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.967931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 418/850] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:10.967931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f933e8-0423-44e5-875f-467d7a02d62b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.972578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 433/851] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:10.975099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.975572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.975973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.983864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.003981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:11.005239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:11.088991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 434/852] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:11.088991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.095648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f933e8-0423-44e5-875f-467d7a02d62b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.145141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.145762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.146021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.146292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.146921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:11.147223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.148580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.156259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 419/853] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:11.218147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8900adf9-6897-495b-955a-9de4ba1a809d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.253021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.260132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.261062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.261339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.261665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.273270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.274772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:11.292071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:11.293107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:11.294016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:11.294016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:11.294016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:11.294016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 435/854] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:11.302239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.333795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.372417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.417038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747ac6ce-d40c-4ff1-a7bc-f990d329823b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.458976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.459571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.459852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.460123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.460767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:11.461093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.462522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.475609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 420/855] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:11.482794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.495387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418db2b-2cc0-4972-b052-a78892bda274 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.532834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.533546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.533823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.534089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.546199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:11.551837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:11.552768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:11.553660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:11.558040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 436/856] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:11.559017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.574599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.591089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b85bc9-00e1-4cec-89d4-7ffecb32a356 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.612509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.625350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.626146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.626146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.626436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.633170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.634805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:11.646410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:11.647539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:11.648660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:11.652838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 421/857] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:11.654195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.677911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63ce405-08dc-43a4-91f4-0bda175ac44e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.715903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.716790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.717874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.718445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.720287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:11.720756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.724217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.742703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 437/858] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:11.744368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.772961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9b93bb-1cc6-455b-9ca4-5ec70c197fcf None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.812931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.813867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.814317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.814759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.815628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:11.816088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.818187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.837450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 422/859] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:11.839171 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.849132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.874147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2bb39-0516-427a-a7a9-efd0f4a5a532 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.883967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.921145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.922022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.922448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.922859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.935364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.937288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:11.952907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:11.954049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:11.954940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:11.954940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:11.954940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:11.957066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 438/860] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:11.963028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb7294-4276-4e22-b4ee-ff9084089c51 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.965466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.009527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.010612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.011117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.011584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.012497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=10649368308f4dc4b8b887319cbcac8f) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:12.013363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.015776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.028071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 423/861] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/role_assignments?user.id=10649368308f4dc4b8b887319cbcac8f&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:12.079569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4af52f-7b9e-4411-b103-f8543644aea7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.122559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.123192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.123478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.123815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.124486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:12.124791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.126214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.142380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 439/862] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:12.150664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.236644 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.274972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0609cd13-4035-477e-96ca-b14d82c13fab None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.309303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.309929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.310210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.310478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.318544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.320099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:12.332036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:12.333021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:12.333987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 424/863] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:12.342019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.362092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.448009 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a65285-3799-4576-b2ec-b5d3c66567ae None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.483698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.484719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.485222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.485696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.486614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:12.487120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.488820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.507345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 440/864] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:12.513807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.620662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d60d5-6b9a-4afc-b90d-8083d3fbc9fc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.650243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.656132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.656749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.657056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.657335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.725701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.727077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.827250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 425/865] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:12.832788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.911750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.935950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc265ee1-1590-46e2-902a-94375ac687af None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.970510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.971149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.971427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.971697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.982536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.983917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:12.999293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:13.000242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:13.001154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:13.001154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:13.001154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:13.001154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 441/866] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:13.006380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.044311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.116168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa1fe52-e90a-4e4e-80b9-8e47a966432e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.156435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.157536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.157949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.158229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.158896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:13.159203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.160601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7266f7d-ec15-47df-a6cf-75743366be9f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.173447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 426/867] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:13.178972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.263609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e210025-f6be-4138-8691-5829931eca18 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.263941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e210025-f6be-4138-8691-5829931eca18 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.264162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e210025-f6be-4138-8691-5829931eca18 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.265826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 427/868] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:13.273491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0274f1-882d-40dc-b1fa-357e7b92d955 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.273871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0274f1-882d-40dc-b1fa-357e7b92d955 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.274781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0274f1-882d-40dc-b1fa-357e7b92d955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.285177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb98f2cb-0247-4e99-870a-01f48b66e4ab None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.295789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2e0274f1-882d-40dc-b1fa-357e7b92d955 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:13.297154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2e0274f1-882d-40dc-b1fa-357e7b92d955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:13.317822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.328366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.329453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.329897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.330365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.338377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.340002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:13.351761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:13.353747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:13.355313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:13.359614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 442/869] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:13.366045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.392986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 428/870] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:13 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:13.412531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf49492-218d-4c89-8013-abe1a48f721d None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.412832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf49492-218d-4c89-8013-abe1a48f721d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.413118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf49492-218d-4c89-8013-abe1a48f721d None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.414573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 429/871] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:13.419789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.420075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.420347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.438933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:13.440123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:13.482593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ddc2b9-7cd5-448f-ae9d-c5e647507af3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.489281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.517746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 430/872] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:13 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:13.517915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.518377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.518695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.518937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.519627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:13.519934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.528907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.533807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 443/873] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:13.540162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.644415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb89398-2e7a-4743-b720-39a70c6f27f3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.679601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.681098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.681660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.682139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.683043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=3546ff5fc9094d35921920f32e0c7476) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:13.683550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.685191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.691153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.692233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 431/874] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/role_assignments?user.id=3546ff5fc9094d35921920f32e0c7476&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:13.749836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.785143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.803419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.808124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0eb86e-a778-4289-859d-8742129eef3d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.846820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06bf46a-dfd4-4234-a07b-13b5ef3fb36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.847448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bf46a-dfd4-4234-a07b-13b5ef3fb36d None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.847730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bf46a-dfd4-4234-a07b-13b5ef3fb36d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.848005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bf46a-dfd4-4234-a07b-13b5ef3fb36d None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.848689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06bf46a-dfd4-4234-a07b-13b5ef3fb36d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.850283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06bf46a-dfd4-4234-a07b-13b5ef3fb36d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.869287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 444/875] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:59:13 2022] POST /identity/v3/services => generated 269 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:13.911816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb2dba-6f6d-4d80-8c4d-4bce404f57c7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.960829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5f078b-ef95-4830-8b5b-ba48f775bbd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.961953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5f078b-ef95-4830-8b5b-ba48f775bbd6 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.962438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5f078b-ef95-4830-8b5b-ba48f775bbd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.962927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5f078b-ef95-4830-8b5b-ba48f775bbd6 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.963905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5f078b-ef95-4830-8b5b-ba48f775bbd6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.965630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5f078b-ef95-4830-8b5b-ba48f775bbd6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.985998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 432/876] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:59:13 2022] POST /identity/v3/services => generated 251 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:14.721077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbefab-729a-4976-969c-465cda5a000c None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:14.721654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbefab-729a-4976-969c-465cda5a000c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:14.721882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbefab-729a-4976-969c-465cda5a000c None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:14.724018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 445/877] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:14.731389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:14.731868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:14.732138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:14.758061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:14.759947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:14.846312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:14.857358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 433/878] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:14 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:15.219694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbefab-729a-4976-969c-465cda5a000c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:15.343210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbefab-729a-4976-969c-465cda5a000c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:15.361209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:15.378536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:15.379763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.380048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.380323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.380933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:15.382369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:15.391381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:15.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:15.393529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 446/879] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:15.399657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:15.505576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188c2d5-fe3f-4dc4-826d-2c3920e5c93d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:15.542834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:15.543775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.544248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.544721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.545563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:15.546059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:15.547640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4669738f-a44d-41c8-88e1-27b397550a69 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:15.553002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 434/880] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:15.559571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:15.665030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b200d1a-0bc0-4120-bb83-b8bec7dfd2f5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:15.701107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:15.701718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.702000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.702273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.702908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:15.703209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:15.704585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2131951-0e05-49e5-8961-f83ca8bd3a1d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:15.709810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 447/881] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:15 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:16.034079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fe864-c058-4e44-bfaa-e228c79507e4 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.035112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fe864-c058-4e44-bfaa-e228c79507e4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.035635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fe864-c058-4e44-bfaa-e228c79507e4 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.037651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 435/882] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:16.044360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3379ef-45de-46be-ad0f-b470d37818a4 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.045355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3379ef-45de-46be-ad0f-b470d37818a4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.045862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3379ef-45de-46be-ad0f-b470d37818a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.064418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7c3379ef-45de-46be-ad0f-b470d37818a4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:16.065987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7c3379ef-45de-46be-ad0f-b470d37818a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:16.096528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.137402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c008d992-7cae-44d1-9073-07207d055cfe None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.138058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c008d992-7cae-44d1-9073-07207d055cfe None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.138527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c008d992-7cae-44d1-9073-07207d055cfe None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.139599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 448/883] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:16 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:16.140243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 436/884] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:16.146251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3539781-ddc3-4d3b-b8a8-a164bf5208ea None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.146749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3539781-ddc3-4d3b-b8a8-a164bf5208ea None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.147193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3539781-ddc3-4d3b-b8a8-a164bf5208ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.166396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-b3539781-ddc3-4d3b-b8a8-a164bf5208ea None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:16.167846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b3539781-ddc3-4d3b-b8a8-a164bf5208ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:16.239546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 449/885] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:16 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:16.391501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31902c5d-1696-48c4-8f63-23b8e94589ba None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.392013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31902c5d-1696-48c4-8f63-23b8e94589ba None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.392443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31902c5d-1696-48c4-8f63-23b8e94589ba None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.394194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 437/886] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:16.403003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.403546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.404041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.422991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:16.423768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31902c5d-1696-48c4-8f63-23b8e94589ba None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.424214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:16.501297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 450/887] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:16 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:16.506301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.522173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.555471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.564224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31902c5d-1696-48c4-8f63-23b8e94589ba None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.619156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.620218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.621500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.621953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.635582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:16.642240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:16.643226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:16.644250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:16.646420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 438/888] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:16.656738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.672203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7224f74a-e8eb-402d-ae09-5d6b6fefbf3b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.722970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.723814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.724094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.724416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.725324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.726697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:16.739160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:16.740393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:16.741526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:16.745440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 451/889] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:16.746360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.786783 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03c76fa-9b8e-455f-9b9e-50f7c143643c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.808975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.834609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.835612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.835934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.836247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.837042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:16.837377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.838777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.849268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 439/890] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:16.850172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.869971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.881268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826d2076-bb1f-42e3-bb0c-1474ecda9c05 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.917273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.917807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.918084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.918358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.919452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:16.919958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.921733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.926542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 452/891] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:16.927245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.968487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f427016-42d5-48bc-9ab2-98a01b256597 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.005764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.006403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.006685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.006960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.007541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.008964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:17.017861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:17.018893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:17.019748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 440/892] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:17.020556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.034603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a0fdff-f515-4fac-9385-446d42ab176b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.069271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.069926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.070176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.070451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.074009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.081464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.082903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:17.097345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:17.098328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:17.099228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:17.099228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:17.099228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:17.099228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 453/893] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:17.099228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.112790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.134681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef618e-a5d3-476d-b31c-2394164aa254 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.169604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.170459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.170828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.171106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.171645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.171954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.173985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.184043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 441/894] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.184043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.225084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6226ab-d58c-40cc-a864-89395ab4e9f5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.260050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.260790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.261100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.261373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.261998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.262300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.263794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.268937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 454/895] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.269823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.296293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed512a-dfcd-426a-ad4c-541ff03d933a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.307808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.335571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.336277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.336602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.336896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.337535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.337832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.339189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.343474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.353587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 442/896] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.359345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.377697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bfb6ba-7117-4d02-ac3d-73d7e3f29ad0 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.415481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.416083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.416351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.416622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.417475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] RBAC: Adding query filter params (interface=public, service_id=b7b4fc3d70e94e9882fd972710c60757, region_id=RegionOne) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.417672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.419031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.428974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 455/897] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/endpoints?service_id=b7b4fc3d70e94e9882fd972710c60757&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.428974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.472410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154e7078-54ee-4504-922f-b3c5565b2754 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.511519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.512126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.512404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.512672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.513383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.513802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.515147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.525305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 443/898] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.525305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.530853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af013f5-5932-451d-8d4f-3f0dc9212e68 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.561648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.565576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.566219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.566760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.567027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.569403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.579124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.580567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:17.597577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:17.598561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 456/899] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:17.599530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.641866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91deae6f-d2db-42cb-a39d-28caeccfb8b0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.677766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.678465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.678746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.679072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.679610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.681501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.696902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 444/900] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/services => generated 1268 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 01:59:17.696902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.710881 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96c70a-d45f-409e-9d24-a52e4c9283ac None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.750712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.751459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.751870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.752234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.752990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] RBAC: Adding query filter params (interface=public, service_id=52486724e9244e72bb813010ee154456, region_id=RegionOne) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.753302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.754711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.763766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 457/901] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/endpoints?service_id=52486724e9244e72bb813010ee154456&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.773268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.813128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.817986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7601f863-461c-4457-b891-43a05352bad7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.845350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.861702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.861958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.862050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.862318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.863075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.863382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.864858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.885500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 445/902] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.892465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.900376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c933fd6d-889c-411e-9bb5-fe0c2b44641c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.949028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041b5ef1-fea6-4195-8020-386e19f14796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.949545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b5ef1-fea6-4195-8020-386e19f14796 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.949823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b5ef1-fea6-4195-8020-386e19f14796 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.950095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b5ef1-fea6-4195-8020-386e19f14796 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.950703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041b5ef1-fea6-4195-8020-386e19f14796 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.952073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041b5ef1-fea6-4195-8020-386e19f14796 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.963472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 458/903] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/services => generated 1268 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 01:59:17.965085 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c7d5b-f274-466b-a957-822a88fd003b None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.965532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c7d5b-f274-466b-a957-822a88fd003b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.965977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c7d5b-f274-466b-a957-822a88fd003b None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.971958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 459/904] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:17 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:17.973799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.974290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.974739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.995382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:17.997038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:18.019208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dde7bc-26c6-4709-9cf1-712872c9022f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.058284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.059020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.059421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.059785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.082709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 460/905] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:17 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:18.094274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.139302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.140764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.262964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 446/906] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:18.267738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.325084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.361542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.374043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7488678-cc1b-47f4-95e5-ddcef26da74c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.408653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.409348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.409625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.409891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.421590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.423064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.425243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:18.442590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:18.448324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:18.449264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:18.449264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:18.449264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:18.449264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 461/907] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:18.451483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.484311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b37332-5aa9-4075-ac2c-691cb9c514e0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.538790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.540081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.541371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.541662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.542445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.543907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.561984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 447/908] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:59:18 2022] POST /identity/v3/services => generated 241 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:18.583159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0bf1e-245e-473b-a59a-36ce32d3c8dc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.590145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.632387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.633496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.634483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.634947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.635803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:18.636332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.637970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.659465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 462/909] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:18.665641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.780923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99087095-d7c0-4fbd-8889-e2faf2f44776 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.798972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.819469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.820104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.820387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.820668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.828106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.829575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:18.841330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:18.842225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:18.843411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 448/910] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:18.850173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.957487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccaa437-0201-43fa-bbae-f6d9d57f2747 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.992938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.993823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.994239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.994667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.995494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:18.995948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.997531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed041758-8fad-429b-9b44-b1828e1a0f28 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.014750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 463/911] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:19.016294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.121452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf46dd01-2a43-42a8-b877-37c39d3d1dfd None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.156297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.157180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.157617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.158036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.158934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] RBAC: Adding query filter params (role.id=c5cd5d31d6504254b5fa4fb776b12311, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=3546ff5fc9094d35921920f32e0c7476) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:19.159399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.160973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f6162-8a80-4cfc-821b-8416e6424ed8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.171568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 449/912] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/role_assignments?user.id=3546ff5fc9094d35921920f32e0c7476&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=c5cd5d31d6504254b5fa4fb776b12311 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:19.969113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14390a0c-e137-4b13-a0d5-8bc41239c780 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.969626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14390a0c-e137-4b13-a0d5-8bc41239c780 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.969626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14390a0c-e137-4b13-a0d5-8bc41239c780 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.971149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 464/913] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:19.976480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ed68aa-0586-4b7f-934f-6cf41a5547bc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.976761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ed68aa-0586-4b7f-934f-6cf41a5547bc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.977092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ed68aa-0586-4b7f-934f-6cf41a5547bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.996110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-73ed68aa-0586-4b7f-934f-6cf41a5547bc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:19.997330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-73ed68aa-0586-4b7f-934f-6cf41a5547bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:20.032997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.075572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 450/914] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:20.221178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ae8d5-d1d6-48d5-a728-aec38821d84f None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.221368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ae8d5-d1d6-48d5-a728-aec38821d84f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.221642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8ae8d5-d1d6-48d5-a728-aec38821d84f None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.223211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 465/915] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:20.228338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.228804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.228934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.249015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:20.250452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:20.320770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 451/916] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:20.343985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8ae8d5-d1d6-48d5-a728-aec38821d84f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.370966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.454581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8ae8d5-d1d6-48d5-a728-aec38821d84f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.493939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.494496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.494777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.495045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.495627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.497025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:20.506360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:20.507289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:20.508158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 466/917] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:20.513853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.599507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.617620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.621087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef63194-ad1d-4424-98df-9ed430317861 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.657762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.658531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.658914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.659286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.660015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:20.660414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.661910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.671861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 452/918] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:20.672917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.725222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.740198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9192469-a648-481d-917a-ef40ad2fe26f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.781245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.782094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.782445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.782790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.783439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.783725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab920ca-926c-4ead-b630-24913dc08915 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.784962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:20.794050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:20.795058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:20.795907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 467/919] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:20.802930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.823484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.824228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.824512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.824786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.825538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:20.825857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.827226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.836985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 453/920] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:20.840196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.878702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.925570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd297f3-4514-4ae9-b982-c7e4bd2438bc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.962352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.962945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.963228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.963804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.964455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:20.964764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.966178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f087c5-4cfb-4b73-82fe-2186829fca81 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.971857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 468/921] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:20.972397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34198b58-c193-49cd-a928-cb476e0a24b0 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.973894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa4eb5-a016-418d-a3e5-b2e8d417706a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.974431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa4eb5-a016-418d-a3e5-b2e8d417706a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.975008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caa4eb5-a016-418d-a3e5-b2e8d417706a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.977178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 469/922] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:20.978804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caa4eb5-a016-418d-a3e5-b2e8d417706a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.994920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.016601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1105f2a5-8d87-4191-a7e0-a77275841dcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.017240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f2a5-8d87-4191-a7e0-a77275841dcf None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.017844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f2a5-8d87-4191-a7e0-a77275841dcf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.018127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1105f2a5-8d87-4191-a7e0-a77275841dcf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.019596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105f2a5-8d87-4191-a7e0-a77275841dcf None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.020982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1105f2a5-8d87-4191-a7e0-a77275841dcf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.045764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 454/923] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:20 2022] POST /identity/v3/endpoints => generated 347 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:21.047359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.047713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.048014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.074067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:21.075925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:21.107346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caa4eb5-a016-418d-a3e5-b2e8d417706a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.146793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.147417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.147698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.147973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.148614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:21.149047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.151023 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.156790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 470/924] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:21.162432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 455/925] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:21.169051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.244173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.278207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1ef0d-58a4-4263-9aed-61d13b720dbd None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.317676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c9ba0-bcb2-497f-b305-1e2cdc814955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.318551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c9ba0-bcb2-497f-b305-1e2cdc814955 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.318974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c9ba0-bcb2-497f-b305-1e2cdc814955 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.319385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c9ba0-bcb2-497f-b305-1e2cdc814955 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.320212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c9ba0-bcb2-497f-b305-1e2cdc814955 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.322159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c9ba0-bcb2-497f-b305-1e2cdc814955 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.356576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 471/926] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:21 2022] POST /identity/v3/endpoints => generated 346 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:21.448592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.455989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576190f-923d-4f71-a250-8f5515e9e31a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.456284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576190f-923d-4f71-a250-8f5515e9e31a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.456534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576190f-923d-4f71-a250-8f5515e9e31a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.457987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 472/927] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:21.464137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.464448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.464690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.466425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.486564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:21.488045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:21.572562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca619a1-72b0-4772-9623-5f8fb293e36e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.577182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 473/928] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:21.619620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.620258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.620534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.620805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.621420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.622769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:21.631784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:21.632792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:21.634234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 456/929] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:21.639648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.699339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.750216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a206bca0-688a-428f-ae63-5215bdec5fa2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.784532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.785171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.785450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.785720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.786407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:21.786711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.788129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.793632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 474/930] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:21.799083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.862405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.897077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.925451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd774d30-62c9-4b6d-9d47-62535f7adc14 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.976042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.976684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.976991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.977789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.978421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:21.978539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3c0554-7936-4e17-8ed9-e3ce846e697b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.978957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.980081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.990541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 457/931] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:21.991911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.999679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.018601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.019238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.019515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.019786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.030846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:22.035680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:22.037563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:22.038667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 475/932] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:22.045961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.118838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e4086c-8a77-4fc5-83f3-a948f3c61cd3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.163168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.163829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.164100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.164419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.165223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] RBAC: Adding query filter params (interface=public, service_id=a257399ca103401089d8f865e948e26d, region_id=RegionOne) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:22.165532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.167740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.175425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 458/933] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/endpoints?service_id=a257399ca103401089d8f865e948e26d&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:22.182984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.188674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3faa11-9e3d-4def-8977-901469486587 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.195457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.229688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.230306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.230585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.230910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.231673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:22.231978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.233482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.247095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 476/934] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:22.248481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.294956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.321362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eff06-eefa-4309-b462-e4ac0bd085cd None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.367515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.368304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.368592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.368992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.369553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.370918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.377278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdae9aa4-f52b-4873-818d-37bc76bfe1e6 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.384561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 459/935] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/services => generated 1497 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 01:59:22.417650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.418759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.419241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.419728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.432780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.434688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:22.451583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:22.452602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:22.453515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:22.453515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:22.453515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:22.455779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 477/936] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:22.462889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.543886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.581867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca68a1-2136-4eaa-ad03-ed44727c1e58 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.619625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.620153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.620428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.620703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.621649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:22.621949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.623318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.636902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 460/937] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:22.642727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.735555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.745448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed1a48-9c32-4346-aa30-3eeeb7e44e60 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.780585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.781240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.781511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.781776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.790097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.791523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:22.802856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:22.803809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:22.804766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 478/938] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:22.809922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.910672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4b7482-7b4b-483c-a32e-cd11332b4a2b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.939933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.947892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.948921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.949588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.949859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.950642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:22.950951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.952359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.970043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 461/939] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:22.975534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.079129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba1dac6-de1b-4da1-8097-f74d6c29d27a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.113555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.114237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.114518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.114788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.126774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.128226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:23.142822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:23.143788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:23.144703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:23.144703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:23.144703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:23.147809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 479/940] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:23.150745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.259482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4be5720-0bb7-41ef-a79c-8cf7f013cabe None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.295906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.296535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.296819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.297120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.297780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:23.298080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.299478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58e5674-2ca3-4973-b2dc-342ee783785e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.313623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 462/941] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:23.319327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.323330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.417536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1f3e1a-712c-4f01-bbe4-5c09e8d7dd75 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.417887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1f3e1a-712c-4f01-bbe4-5c09e8d7dd75 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.418085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1f3e1a-712c-4f01-bbe4-5c09e8d7dd75 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.419617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 463/942] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:23.424572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.424855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.425160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.431562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2626e8-80d0-45f9-976e-2d6ceda70e49 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.444094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:23.445487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:23.465537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.466402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.466857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.467277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.474043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.475702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:23.488795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:23.489918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:23.491012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:23.493117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 480/943] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:23.493505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.498766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.524407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 464/944] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:23.601262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.613313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c29203-a7b5-4266-af19-c4e28cdf1f7b None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.646520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.647401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.647849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.648348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.649811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:23.650305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.651894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4e2d47-dcad-4517-921c-22a8261272d6 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.664464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 481/945] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:23.670833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.685393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c49b86-6a3b-41f1-ad3a-cfde62fcf0bd None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.685691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c49b86-6a3b-41f1-ad3a-cfde62fcf0bd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.685941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c49b86-6a3b-41f1-ad3a-cfde62fcf0bd None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.688215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 482/946] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:23.694304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.694584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.694856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.718181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:23.720000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:23.783200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8f2f47-f4b1-4d32-aa03-1fc07dd6dc00 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.800816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 483/947] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:23.801683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.838355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.839025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.839344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.839626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/3546ff5fc9094d35921920f32e0c7476/roles/c5cd5d31d6504254b5fa4fb776b12311` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.842149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=3546ff5fc9094d35921920f32e0c7476, role_id=c5cd5d31d6504254b5fa4fb776b12311)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.844541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.861914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 465/948] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:23 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/3546ff5fc9094d35921920f32e0c7476/roles/c5cd5d31d6504254b5fa4fb776b12311 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:23.872630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.923880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60121bf5-8e7f-4d99-9144-d954ce9ed0e6 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.963016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.964112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.964591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.965131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.965943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.967608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:23.977704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:23.978781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:23.979992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:23.982756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 484/949] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/services/compute => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:23.994391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.038866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.090181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.109459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcb6a2-4ff4-4d92-9c12-22ca88dde0c9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.153954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.154835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.155121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.155461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.156193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] RBAC: Adding query filter params (name=compute) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:24.156608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.158011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.169016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 466/950] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:24.170832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.196904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdda7c-2032-49fa-966d-2afe2f0cb354 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.214571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.233016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.233773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.234149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.234515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.235264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.236791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.239507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING keystone.server.flask.common [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 29 01:59:24.258027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.sql.core [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 29 01:59:24.258027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 29 01:59:24.258027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [parameters: {'id': 'e2fa3c94f6fd4facbc77dc995c3846af', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 29 01:59:24.258027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81544) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e2fa3c94f6fd4facbc77dc995c3846af.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e2fa3c94f6fd4facbc77dc995c3846af. Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:59:24.266286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result.read() Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 29 01:59:24.267130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application yield resource Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.session.rollback() Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.force_reraise() Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise self.value Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application session.commit() Sep 29 01:59:24.269089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._prepare_impl() Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.session.flush() Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._flush(objects) Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application compat.raise_( Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application flush_context.execute() Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 29 01:59:24.271262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rec.execute(self) Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 29 01:59:24.278068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result.read() Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:59:24.280122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [parameters: {'id': 'e2fa3c94f6fd4facbc77dc995c3846af', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:24.282308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:24.293701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e2fa3c94f6fd4facbc77dc995c3846af. Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 485/951] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Sep 29 01:59:24 2022] POST /identity/v3/projects => generated 262 bytes in 178 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:24.295891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.297261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddbb621-33fd-45cd-8b15-e27e514b3e4f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.297261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.335827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.336652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.337094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.337438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.338452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:24.338825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.340355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.350896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 467/952] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:24.383641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4633292-a382-482a-bb29-867eba832ff2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.422493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.423134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.423409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.423673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.431418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.433213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-a0436b96-fdde-43b8-a9a1-36fca6c66248 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:24.446000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:24.448213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:24.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 486/953] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:24.456897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.562232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0d0388-2123-462f-830e-9c6cf3817500 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.607442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.608115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.608400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.608672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.609423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:24.609731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.611152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffb4e7-ec74-48a1-8169-23effbf63277 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.630357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 468/954] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:24.757846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518c103-48e6-43cb-a163-663eb940a692 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.758186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518c103-48e6-43cb-a163-663eb940a692 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.758930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9518c103-48e6-43cb-a163-663eb940a692 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.763633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 487/955] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:24 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:24.769428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.769708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.769999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.790263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:24.791497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:24.814321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.870783 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 469/956] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:25.137927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518c103-48e6-43cb-a163-663eb940a692 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.175076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.248333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9518c103-48e6-43cb-a163-663eb940a692 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.282837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.283474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.283755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.284026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.284617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.286027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:25.295063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:25.297668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:25.299715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 488/957] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:25.307108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.409129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d742c451-4f86-4458-9d3d-82ef768f739e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.445089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.446095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.446624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.447109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.447939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:25.448444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.450051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.455816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 470/958] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:25 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:25.461130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.561014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.565828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9c8d4b-692d-4155-95b8-becb7fde7117 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.603828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.604445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.604727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.605031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.605667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:25.605970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.607336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c3784c-f568-44f4-ac12-02ba7db048ad None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.612695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 489/959] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:25 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:25.618532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.644224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.725548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5942f599-cfd8-436f-9922-8a6e33e42558 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.760816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642726fe-cc2e-4612-a944-ff5a66ceb249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.761461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642726fe-cc2e-4612-a944-ff5a66ceb249 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.761771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642726fe-cc2e-4612-a944-ff5a66ceb249 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.762050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642726fe-cc2e-4612-a944-ff5a66ceb249 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.762721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642726fe-cc2e-4612-a944-ff5a66ceb249 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.764078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642726fe-cc2e-4612-a944-ff5a66ceb249 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.787245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 471/960] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:25 2022] POST /identity/v3/endpoints => generated 349 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:26.198588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dace5eb-dd04-4180-b2c3-bcd314d8b6c3 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.199132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dace5eb-dd04-4180-b2c3-bcd314d8b6c3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.199132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dace5eb-dd04-4180-b2c3-bcd314d8b6c3 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.200692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 490/961] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:26.205941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b776886b-1051-41e5-a79d-ffc31a51881a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.206227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b776886b-1051-41e5-a79d-ffc31a51881a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.206496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b776886b-1051-41e5-a79d-ffc31a51881a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.225016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b776886b-1051-41e5-a79d-ffc31a51881a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:26.226193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b776886b-1051-41e5-a79d-ffc31a51881a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:26.303857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 472/962] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:26.575904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dace5eb-dd04-4180-b2c3-bcd314d8b6c3 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.680639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dace5eb-dd04-4180-b2c3-bcd314d8b6c3 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.681005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00bcbaf-9df0-430a-b323-6a456e3875ee None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.681519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00bcbaf-9df0-430a-b323-6a456e3875ee None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.681519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00bcbaf-9df0-430a-b323-6a456e3875ee None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.683050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 473/963] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:26.688929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.689243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.689482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.696405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:26.711046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:26.713771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:26.718058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.718685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.718960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.719228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.729559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:26.734340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:26.736538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:26.738622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 491/964] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:26.744890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.773234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:26.790874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 474/965] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:26.843058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd31b53e-fcef-48d0-8a00-525bf134aefc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.881654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.882259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.882532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.882800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.883544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:26.883846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:26.885230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b550db-684d-4480-8554-d71b531a5380 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:26.893873 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 492/966] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:26.900152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.968224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ef072-c915-4ec1-a739-8ad7613445ae None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.968552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ef072-c915-4ec1-a739-8ad7613445ae None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.968762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ef072-c915-4ec1-a739-8ad7613445ae None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.970231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 493/967] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:26.975199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.975476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.975742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.995216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:26.996757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:27.007655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d41c42-de1c-406f-97e1-bdee23c3d3a6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.038213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.043973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.044907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.045286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.045649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.059725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.061197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.073433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 494/968] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:27.078183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:27.079112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:27.080012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:27.080012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:27.080012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:27.081990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 475/969] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:27.088263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.096495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.106015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.203341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166c65bd-5587-4b46-b2f6-b2a1d2e4c167 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.223388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e031b9d-2e44-4ce6-bcb1-b35040ddc9d1 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.239995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.240602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.240907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.241181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.241820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.243594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.265249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 476/970] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:59:27 2022] POST /identity/v3/services => generated 242 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:27.276571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.277667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.278154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.278648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.279664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:27.280233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.281948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.300954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 495/971] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:27.303176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.356178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.377852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.421231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9770a5c-af93-495e-bba2-151cd13f2ee9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.457127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.457756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.458034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.458303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.466678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.467562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.468107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:27.480180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:27.481199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:27.482142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 477/972] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:27.488947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.511855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2431b3c-68a5-4093-9822-820a0f89bd1c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.546362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.546967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.547241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.547508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.548153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.549624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.568304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.sql.core [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 29 01:59:27.568304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 29 01:59:27.568304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [parameters: {'user_id': '9bfe8b8628aa435bbc1cee3d29729da8', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 29 01:59:27.568304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81544) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result.read() Sep 29 01:59:27.577733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:59:27.578661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application yield resource Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.session.rollback() Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.force_reraise() Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise self.value Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application session.commit() Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 29 01:59:27.579602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._prepare_impl() Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.session.flush() Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._flush(objects) Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application compat.raise_( Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application flush_context.execute() Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rec.execute(self) Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 29 01:59:27.580486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:59:27.581449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application result.read() Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:59:27.582302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [parameters: {'user_id': '9bfe8b8628aa435bbc1cee3d29729da8', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:27.583148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 29 01:59:27.583952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 496/973] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Sep 29 01:59:27 2022] POST /identity/v3/users => generated 160 bytes in 202 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:27.589793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.613522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2017fa-3c24-478f-a804-9f3094ab9b1c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.616107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.656435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.657193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.657654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.657934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.658997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:27.659127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.660489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.686510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 478/974] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:27.687789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.706291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851aa6e-37ae-4a4a-8b7e-3c0362f7b13f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.744696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.751919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.752831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.753417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.753883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.763176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.765061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:27.777109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:27.778130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:27.779102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:27.781737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 497/975] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/users/cinder => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:27.792928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.801014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e61b665-b496-4412-9481-36586ad6fa31 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.839482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f297b24b-cf9e-4465-8421-4283d6d2a8f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.840102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f297b24b-cf9e-4465-8421-4283d6d2a8f5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.840390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f297b24b-cf9e-4465-8421-4283d6d2a8f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.840668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f297b24b-cf9e-4465-8421-4283d6d2a8f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.876038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.917781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f297b24b-cf9e-4465-8421-4283d6d2a8f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.919303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f297b24b-cf9e-4465-8421-4283d6d2a8f5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.923994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0551ee1d-497c-4d2f-8a63-dd8e67e3b2b3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.962046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.962934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.963352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.963799 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.964655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:27.965160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.966789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.013290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 498/976] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.036591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 479/977] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:28.038239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.065319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773be59a-dc43-4aaa-93d5-f6a17c0451ec None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.065636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773be59a-dc43-4aaa-93d5-f6a17c0451ec None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.065876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773be59a-dc43-4aaa-93d5-f6a17c0451ec None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.067855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 480/978] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:28.073574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.073919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.074128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.093259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:28.094500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:28.116198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.175038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.187079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3501891a-9429-4d9b-b3c3-0a58a3e5d212 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.187994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 481/979] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:28 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:28.226074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.226771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.227057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.227333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.238534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.240268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:28.257412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:28.258574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:28.259608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:28.259608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:28.259608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:28.259608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 499/980] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:28.264537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.369741 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857d07a-3e45-4768-8ba6-5efe49c6a4e0 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.407637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.408503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.408959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.413200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.414045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.414579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.417205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.429668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 482/981] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.436107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.470860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.526054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.539533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fcf65d-8ad2-4aa8-bac9-ef338196069c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.542115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.581200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.581647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e3ebea-cacb-4835-9ccb-1097d0e3941c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.582421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.582918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.583574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.593216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.594855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:28.607721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:28.608621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:28.609594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 500/982] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:28.614611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.620589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.621326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.621673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.622013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.622675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.624110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:28.633175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:28.634696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:28.635540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 483/983] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:28.640491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.728617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ebbfd-7c23-4b22-8a2d-3683dfa4ca49 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.751456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f784ea2b-2854-4ab7-8b63-4373e0705092 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.763666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.764594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.764967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.765442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.766566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.766945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.768630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.768928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.790776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 501/984] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.790941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.790941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.790941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.790941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.791289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.791289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.791289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.792464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.794960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 484/985] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:28.800372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.825867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.900214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2c2cec-cfc1-48e6-9d1e-ef847aaee87b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.913295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af24025-6f2e-4710-8e05-f760c52b96d9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.937794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.938399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.938678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.938946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.939561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.939862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.941482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.946788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 485/986] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:28.952425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.953832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.954431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.954702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.954973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.955687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] RBAC: Adding query filter params (role.id=c5cd5d31d6504254b5fa4fb776b12311, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=3546ff5fc9094d35921920f32e0c7476) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.955995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.957456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.964058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 502/987] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/role_assignments?user.id=3546ff5fc9094d35921920f32e0c7476&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=c5cd5d31d6504254b5fa4fb776b12311 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:29.002099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.062017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7b1370-d1b2-44fa-b86a-90aeb80e94d8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.098376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.099077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.099425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.099764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.100557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] RBAC: Adding query filter params (interface=admin, service_id=a257399ca103401089d8f865e948e26d, region_id=RegionOne) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:29.101010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.102505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c83da-a94d-4439-9466-5bd61d9e312e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.112967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 486/988] 10.222.0.27 () {56 vars in 1249 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/endpoints?service_id=a257399ca103401089d8f865e948e26d&interface=admin®ion_id=RegionOne => generated 193 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:29.114686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.208863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.224876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63147b34-effa-4eee-8de9-822d677b2721 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.260626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3db75-125e-40f4-94c4-bf01c3f65d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.261252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3db75-125e-40f4-94c4-bf01c3f65d7f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.261485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3db75-125e-40f4-94c4-bf01c3f65d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.261754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3db75-125e-40f4-94c4-bf01c3f65d7f None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.262359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc3db75-125e-40f4-94c4-bf01c3f65d7f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.263728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc3db75-125e-40f4-94c4-bf01c3f65d7f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.270449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 503/989] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/services => generated 1727 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 01:59:29.562109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bace99-9419-487e-93e0-f0f21e1c5adf None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.562313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bace99-9419-487e-93e0-f0f21e1c5adf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.562573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bace99-9419-487e-93e0-f0f21e1c5adf None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.563931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 487/990] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:29.569455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.569741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.570012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.590990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:29.592770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:29.660961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 504/991] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:29 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:29.930827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bace99-9419-487e-93e0-f0f21e1c5adf None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:30.033480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bace99-9419-487e-93e0-f0f21e1c5adf None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:30.042613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:30.068695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:30.069347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:30.069624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:30.069894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:30.070462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:30.071849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:30.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:30.081430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:30.082269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:30.083854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 488/992] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/services/compute => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:30.087324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:30.176098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:30.197280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc48f6f-6da8-430e-8252-f06ef3167f15 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:30.230780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:30.231386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:30.231666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:30.231935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:30.232553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] RBAC: Adding query filter params (name=compute) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:30.232852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:30.234258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:30.239187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 505/993] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:30.245391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:30.343606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7282ed8-de6b-4ed2-9fae-ddb8bd6123e9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:30.376437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:30.377068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:30.377352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:30.377628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:30.378249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:30.378554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:30.379910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:30.384843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 489/994] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:30 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:30.390654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:30.493302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40197d1-dfd1-4968-b63d-acf99af2d0af None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:30.527054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:30.527659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:30.527941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:30.528214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:30.528939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] RBAC: Adding query filter params (interface=public, service_id=e3f2a053e2d44709a9a95d11ace36815, region_id=RegionOne) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:30.529352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:30.530719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2366b21-641d-4e9b-9e20-4901ae140f52 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:30.535864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 506/995] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:30 2022] GET /identity/v3/endpoints?service_id=e3f2a053e2d44709a9a95d11ace36815&interface=public®ion_id=RegionOne => generated 194 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:30.541552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:30.581205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:30.649738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12060386-a978-4665-88b6-00befbe19d63 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:30.683294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150851ba-f931-49d1-b929-a55954255604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:30.683896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150851ba-f931-49d1-b929-a55954255604 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:30.684177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150851ba-f931-49d1-b929-a55954255604 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:30.684449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150851ba-f931-49d1-b929-a55954255604 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:30.685071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150851ba-f931-49d1-b929-a55954255604 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:30.686450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150851ba-f931-49d1-b929-a55954255604 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:30.692987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 490/996] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:30 2022] GET /identity/v3/services => generated 1727 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 01:59:31.150556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0d107-0fba-474f-b497-8b2f9e4c53c4 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.150962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0d107-0fba-474f-b497-8b2f9e4c53c4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.151156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e0d107-0fba-474f-b497-8b2f9e4c53c4 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.152616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 507/997] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:31.157640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0024c66-7067-4a74-bfb8-f7b4e4d434da None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.158079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0024c66-7067-4a74-bfb8-f7b4e4d434da None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.158261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0024c66-7067-4a74-bfb8-f7b4e4d434da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.175986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c0024c66-7067-4a74-bfb8-f7b4e4d434da None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:31.177208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c0024c66-7067-4a74-bfb8-f7b4e4d434da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:31.245083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 491/998] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:31 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:31.443260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c80c2-ff77-4d41-af01-94c070e4c106 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.443743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c80c2-ff77-4d41-af01-94c070e4c106 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.443743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1c80c2-ff77-4d41-af01-94c070e4c106 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.445256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 508/999] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:31.450282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.450575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.450851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.469694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:31.470889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:31.494448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:31.507567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1c80c2-ff77-4d41-af01-94c070e4c106 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:31.518761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:31.540293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 492/1000] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:31 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:31.617568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1c80c2-ff77-4d41-af01-94c070e4c106 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:31.651798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:31.652421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.652695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.652992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] PATH_INFO: `/v3/services/image` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.653562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:31.654900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:31.663422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:31.664421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:31.665795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:31.669198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 509/1001] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Sep 29 01:59:31 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:31.670832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:31.770946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18664320-4007-4de9-bf94-5c9df9ca3314 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:31.803446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:31.805755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:31.806397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.806641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.806910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.807542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] RBAC: Adding query filter params (name=image) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:31.807844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:31.809222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:31.814176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 493/1002] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:31 2022] GET /identity/v3/services?name=image => generated 122 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:31.819891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:31.848585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:31.876408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:31.909094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d41884-4fe0-4070-bd3e-defc4730e517 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:31.936013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34bd824-8d52-4528-85bb-dd3914e60f91 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:31.944427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:31.945082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.945371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.945623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.946778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:31.948131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:31.957382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:31.958468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:31.959313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 510/1003] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Sep 29 01:59:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:31.964240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:31.975258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:31.975872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:31.976157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:31.976431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:31.977117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] RBAC: Adding query filter params (type=image) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:31.977424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:31.978784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:31.993109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 494/1004] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:31 2022] GET /identity/v3/services?type=image => generated 122 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:32.082300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd75490-1f96-4a24-9b09-60cbd947f269 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:32.116631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:32.117281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.118066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.118321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.118957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:32.119258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:32.120645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:32.122824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:32.127609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 511/1005] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:31 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:32.133895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:32.226870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:32.240169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61d2c0-db99-4143-8d75-fa61b3a38f44 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:32.275277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:32.275882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.276162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.276434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.277085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:32.277393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:32.278749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d0014a-6213-43a2-819c-d7cdb0a21d40 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:32.283824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 495/1006] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Sep 29 01:59:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:32.289322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:32.390185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0887e0-c7f2-4b85-969e-39f4932fa63a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:32.424247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d0de07-82cd-4f44-948e-2d80289d457f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:32.424908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d0de07-82cd-4f44-948e-2d80289d457f None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.425192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d0de07-82cd-4f44-948e-2d80289d457f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.425468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d0de07-82cd-4f44-948e-2d80289d457f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.426139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d0de07-82cd-4f44-948e-2d80289d457f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:32.427493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d0de07-82cd-4f44-948e-2d80289d457f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:32.452910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 512/1007] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:32 2022] POST /identity/v3/endpoints => generated 325 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:32.896097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc510d-5cc3-4245-8255-67bdaac00272 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.896478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc510d-5cc3-4245-8255-67bdaac00272 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.896676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc510d-5cc3-4245-8255-67bdaac00272 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.898035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 496/1008] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:32.903642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.903924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.904192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.922475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:32.923636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:32.980785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.000343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 513/1009] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:32 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:33.267749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc510d-5cc3-4245-8255-67bdaac00272 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.373418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc510d-5cc3-4245-8255-67bdaac00272 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.406473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.407092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.407370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.407638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] PATH_INFO: `/v3/services/compute` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.408209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.409579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:33.418633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:33.419607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:33.420447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 497/1010] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/services/compute => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:33.425568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.524548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08e1b30-f5d8-46a9-bc5e-d631466b9fbb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.557700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.558301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.558579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.558849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.559470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:33.559771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.561164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.571426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 514/1011] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:33.572765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.591928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.672998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16eafe6-dca0-4211-b4dd-666937d024ce None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.707008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.707610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.707886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.708152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.708769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:33.709095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.710438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f14402-6664-4686-84c9-489c60cd2ef4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.715482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 498/1012] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:33.720889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.773142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.826235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333092c3-1d39-4628-83fa-b5dda9d47ce0 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.860757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6777e4be-7164-4afc-8551-d111549ca2f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.861382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777e4be-7164-4afc-8551-d111549ca2f9 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.861658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777e4be-7164-4afc-8551-d111549ca2f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.861926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777e4be-7164-4afc-8551-d111549ca2f9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.862586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6777e4be-7164-4afc-8551-d111549ca2f9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.863938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6777e4be-7164-4afc-8551-d111549ca2f9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.884618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 515/1013] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:33 2022] POST /identity/v3/endpoints => generated 334 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:34.152770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60daf90-bc0c-42be-bdad-8e423bd36dee None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.152770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60daf90-bc0c-42be-bdad-8e423bd36dee None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.152770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60daf90-bc0c-42be-bdad-8e423bd36dee None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.152770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 499/1014] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:34.158241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e88a8b8-8c15-4e74-a0b4-24a18fd493ea None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.158528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e88a8b8-8c15-4e74-a0b4-24a18fd493ea None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.158801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e88a8b8-8c15-4e74-a0b4-24a18fd493ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.177110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7e88a8b8-8c15-4e74-a0b4-24a18fd493ea None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:34.178322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7e88a8b8-8c15-4e74-a0b4-24a18fd493ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:34.252400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 516/1015] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:34.518649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60daf90-bc0c-42be-bdad-8e423bd36dee None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:34.609480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:34.630865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60daf90-bc0c-42be-bdad-8e423bd36dee None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.663134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aad450-593b-4446-ae11-5be38ae32afd None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.664182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aad450-593b-4446-ae11-5be38ae32afd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.665256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aad450-593b-4446-ae11-5be38ae32afd None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.666078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.666678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.666956 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.667227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.667900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 517/1016] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:34.668019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.669385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.672653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.672964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.673240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.688914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 500/1017] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/services => generated 242 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:34.695958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:34.697235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:34.727654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:34.782261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 518/1018] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:35.049884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:35.159908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8003ae94-de30-4ad9-9a18-b3e2d69e8ec8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:35.196441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be6ae5b-b95d-4460-ba54-68343e09e8aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:35.197494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be6ae5b-b95d-4460-ba54-68343e09e8aa None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:35.198012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be6ae5b-b95d-4460-ba54-68343e09e8aa None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:35.198498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be6ae5b-b95d-4460-ba54-68343e09e8aa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:35.199789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be6ae5b-b95d-4460-ba54-68343e09e8aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:35.201565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be6ae5b-b95d-4460-ba54-68343e09e8aa None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:35.203698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 501/1019] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:35.209066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:35.308823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2840ea-bc19-4329-aa4e-50ebc113d356 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:35.341694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573815bc-3c4c-4c29-9117-8e3bd3a3a6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:35.342295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573815bc-3c4c-4c29-9117-8e3bd3a3a6ca None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:35.342571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573815bc-3c4c-4c29-9117-8e3bd3a3a6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:35.342842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573815bc-3c4c-4c29-9117-8e3bd3a3a6ca None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:35.343484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573815bc-3c4c-4c29-9117-8e3bd3a3a6ca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:35.344940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573815bc-3c4c-4c29-9117-8e3bd3a3a6ca None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:35.376896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 519/1020] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 01:59:35 2022] POST /identity/v3/projects => generated 311 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:36.178857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ea8f8-88f4-46a0-968e-63ffe2e1dbf8 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.179423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ea8f8-88f4-46a0-968e-63ffe2e1dbf8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.179423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ea8f8-88f4-46a0-968e-63ffe2e1dbf8 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.180948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 502/1021] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:36.186329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.186794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.186885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.205587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:36.206848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:36.209578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:36.276145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 520/1022] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:36 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:36.558298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ea8f8-88f4-46a0-968e-63ffe2e1dbf8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:36.570669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:36.669710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ea8f8-88f4-46a0-968e-63ffe2e1dbf8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:36.704266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c285bdf9-c088-4b55-bf07-0d5e1f32821b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:36.704916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c285bdf9-c088-4b55-bf07-0d5e1f32821b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.705199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c285bdf9-c088-4b55-bf07-0d5e1f32821b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.705469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c285bdf9-c088-4b55-bf07-0d5e1f32821b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.716619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c285bdf9-c088-4b55-bf07-0d5e1f32821b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-c285bdf9-c088-4b55-bf07-0d5e1f32821b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:36.721364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:36.722297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:36.723140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 503/1023] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 29 01:59:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:36.730378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:36.835691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dbd19c-0891-4ff0-bc68-fe673cf4281e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:36.858130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a48ada-051a-48e9-8525-a00687fad80e None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.858459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a48ada-051a-48e9-8525-a00687fad80e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.858690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a48ada-051a-48e9-8525-a00687fad80e None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.860136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 504/1024] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:36.865283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.865573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.865843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.877557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:36.878182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.878463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.878738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.879413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:36.882209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:36.884126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:36.888458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:36.889681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:36.893199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 521/1025] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 01:59:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:36.898462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:36.901726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:36.967852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:36.975428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 505/1026] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:36 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:37.010570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2631ef75-d1d9-4784-b32d-cb49ff3e0316 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.049373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.050347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.050848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.051320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.063325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.065195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:37.080074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:37.081053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:37.081999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:37.081999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:37.081999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:37.083774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 522/1027] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:37.089987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.190679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7522592a-68b8-4634-bb83-ff9e01f97c84 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.237921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.238884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.239169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.239441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.240089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.240389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.241789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.258201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 506/1028] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:37.258406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.259063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.322906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.325860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.373367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afa2f5f-f85d-4c4b-8198-9d7a75a00a6b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.378413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710bc431-c80c-4f34-a3f5-ffad7a04f28a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.411029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.411647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.411930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.412202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] PATH_INFO: `/v3/users/nova` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.414392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.415559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.415829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.416104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] PATH_INFO: `/v3/services/image` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.416685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.418139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.421057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.422490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:37.427103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:37.428111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:37.428983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:37.433094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 523/1029] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/services/image => generated 86 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:37.434166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:37.436088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:37.437072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 507/1030] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:37.443218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.556814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a61fef-b81b-4cca-ae63-deb80e06aea4 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.559792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d3b9ec-d356-4ea7-8018-d4f96530caca None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.579479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.581225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.602546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.603242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.603517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.603782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.604475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] RBAC: Adding query filter params (name=image) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.604775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.605993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.606198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.606600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.606877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.607267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.607989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.608292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.609734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20444bb8-7bd7-41e4-8c19-8b767cbaba5b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.611860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 524/1031] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/services?name=image => generated 122 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:37.617370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.633936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 508/1032] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:37.635159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7df6c8-dd01-46ee-a8c6-7132154404d6 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.635445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7df6c8-dd01-46ee-a8c6-7132154404d6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.635720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7df6c8-dd01-46ee-a8c6-7132154404d6 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.636944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 509/1033] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:37.640145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7df6c8-dd01-46ee-a8c6-7132154404d6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.716776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd10dc3-ab3f-4167-a64f-808145ef6b6d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.744712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7df6c8-dd01-46ee-a8c6-7132154404d6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.750303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.750902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.751180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.751446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.752067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] RBAC: Adding query filter params (type=image) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.752361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.753744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7925f8-1841-4eeb-ac2a-b3c588594c90 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.758518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 525/1034] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/services?type=image => generated 350 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:37.759856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.760140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.760441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.777656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:37.778822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:37.780174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.780779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.781087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.781360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.822685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.825839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.848409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 526/1035] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:37 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:37.848945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.849327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.850574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.957507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da585f78-e2c7-4bb6-b490-91667e4394c3 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.960712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 510/1036] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 29 01:59:37.966557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.992899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.993502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.993892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.994058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.994747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] RBAC: Adding query filter params (interface=public, service_id=ee7b4c710d5540ab84f47bde5c92451b, region_id=RegionOne) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.995049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.996410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.005866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 527/1037] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/endpoints?service_id=ee7b4c710d5540ab84f47bde5c92451b&interface=public®ion_id=RegionOne => generated 194 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.006791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.049099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.052038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.078564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd1f5e-2043-4290-aa5f-1172e79c7f9f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.115278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.115904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.116182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.116452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.119920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a40a0-2e96-4e6a-adf9-d023a9861aac None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.127118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.128542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:38.143297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.144323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:38.145260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:38.145260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:38.145260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:38.148943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 511/1038] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:38.149753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.156587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.157223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.157503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.157777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.158376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.159736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.166328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 528/1039] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/services => generated 1957 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 01:59:38.167094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.275418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327a529a-da9a-445a-a01a-c1f1d12a9e44 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.279229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9ab76f-9a16-4d4f-9a21-566161e7e97e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.282555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.305234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.315056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.315706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.315982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.316251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.317683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.319127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.324927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 512/1040] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.325131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.325131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.325131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.325131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.325414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.325414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.325414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.326997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.338207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 529/1041] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.343936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.433779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902365c-4c77-4c34-a814-01a5bbc66b10 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.450310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a498a5-775e-43f1-a5d8-532791c549cc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.469933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.470695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.470978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.471248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.471890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.473571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.483903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.484531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.484807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.485104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.492240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.494113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.502780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 513/1042] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:38 2022] POST /identity/v3/users => generated 285 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:38.505779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.506687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:38.507582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:38.512661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 530/1043] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:38.514153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.579604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.635805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdffea9b-7135-4997-abf0-2dbbf0eaa32a None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.672936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.673558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.673837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.674111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.674803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.675114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.676537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841e733f-ab72-4521-a908-e0df7c3b9849 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.690435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 514/1044] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.696752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.726080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.804786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4c8120-3c95-4320-852c-8dc491e39010 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.839323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.839850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.840148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.840413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.841160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] RBAC: Adding query filter params (role.id=c5cd5d31d6504254b5fa4fb776b12311, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=e528956ee4704ac0b5ce4c62826b71ee) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.841462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.842893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca62e15-034d-4db7-9039-76a2c87dbecf None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.856089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 531/1045] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/role_assignments?user.id=e528956ee4704ac0b5ce4c62826b71ee&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=c5cd5d31d6504254b5fa4fb776b12311 => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:40.351383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e4db06-888a-44d5-bf21-a57e23a07f0b None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.351990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e4db06-888a-44d5-bf21-a57e23a07f0b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.351990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e4db06-888a-44d5-bf21-a57e23a07f0b None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.354149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 515/1046] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:40.359969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70cd27a-48f5-4a03-a931-ff394591bfa0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.360449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70cd27a-48f5-4a03-a931-ff394591bfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.360540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70cd27a-48f5-4a03-a931-ff394591bfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.378793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d70cd27a-48f5-4a03-a931-ff394591bfa0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:40.379990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d70cd27a-48f5-4a03-a931-ff394591bfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:40.447091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 532/1047] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:40 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:40.721302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e4db06-888a-44d5-bf21-a57e23a07f0b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.730569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb06821-0d81-44d8-9e2e-c06d2c26d906 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.730861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb06821-0d81-44d8-9e2e-c06d2c26d906 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.731131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb06821-0d81-44d8-9e2e-c06d2c26d906 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.732491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 533/1048] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:40.737703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.737990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.738265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.756625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:40.757843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:40.771563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.830489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 534/1049] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:40 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:40.834181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e4db06-888a-44d5-bf21-a57e23a07f0b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.849210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.872005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.872750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.873073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.873337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] PATH_INFO: `/v3/services/image` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.873932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.875334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-30aa0840-0b90-41de-a1a2-f4b5bebf8524 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:40.885232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:40.886324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:40.890023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 516/1050] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/services/image => generated 86 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:40.893127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.995092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880b5c69-1314-4752-b9b1-b834c0c3d462 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.996433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfd922-89a0-4e2f-a8a4-b5b680affd10 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.996723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfd922-89a0-4e2f-a8a4-b5b680affd10 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.997032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfd922-89a0-4e2f-a8a4-b5b680affd10 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.998494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 517/1051] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:41.003213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.004014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.004326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.023390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:41.025107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:41.031303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.031914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.032220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.032578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.033232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] RBAC: Adding query filter params (name=image) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.033533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.035164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.045945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 535/1052] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:41.046903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.061854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.101307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 518/1053] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:41 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:41.102714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.155467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec210ee-a599-4833-8d3c-793fcdc7a5af None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.192563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.193409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.193689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.193958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.194589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] RBAC: Adding query filter params (type=image) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.194888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.196645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.207395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 536/1054] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/services?type=image => generated 350 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:41.207583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8071d58a-0903-4efb-afbe-f01c7e397b02 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.208458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.216387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.246122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.246743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.247028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.247301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.257934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:41.263173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.264103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:41.265424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 519/1055] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.270005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.298011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.322572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222415c8-ea19-41ba-9d62-6e24ec7399b7 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.358056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.358658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.358932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.359200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.359860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.361248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.377397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e33cb8-f3d2-4a29-82d4-76d400340dcc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.382719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 537/1056] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 01:59:41 2022] POST /identity/v3/endpoints => generated 327 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:41.383823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.430872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.431875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.432352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.432844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.433747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.434248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.435909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.445914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 520/1057] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:41.452187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.481612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.513707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626a67c9-23ae-4802-892a-721e9b782d03 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.544900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.552073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.553609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.553892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.554165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.566408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab3cbc4-e571-4587-ab85-f845ed935f96 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:41.576012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.577064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:41.578535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 538/1058] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.583273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.610499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.611333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.611464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.611671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] PATH_INFO: `/v3/users/9d403b95179b41c587682dbf53335624` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.627348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] RBAC: Authorizing `identity:get_user(user_id=9d403b95179b41c587682dbf53335624)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.628804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.631201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 521/1059] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/users/9d403b95179b41c587682dbf53335624 => generated 285 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:41.637554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.696704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21d495-4c8e-4363-a164-09e21faf886e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.728771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.731202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.731827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.732116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.732392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.733089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.733391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.734765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.743389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 539/1060] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:41.748834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.755697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50e1cc-d902-42bd-b2ee-a31d7f093c21 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.792951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.793486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.793767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.794072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.805648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.852522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d195c23-51f6-4485-bd28-104013d698a1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.869732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.871187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.886197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.886845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.887230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.887505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.900103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.901583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:41.916005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.916998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:41.923041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:41.923041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:41.923041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:41.923041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 540/1061] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.923041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.975608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.998134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 522/1062] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 29 01:59:42.003982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.028632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e405966-d9d7-4929-871d-cb4ddd7ca28a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.035724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.064554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.065110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.065394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.065667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.066305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.066615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.068246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.080597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 541/1063] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:42.086100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.112327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09900e6b-0f44-4b0f-be22-a422364368fa None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.153348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.153987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.154276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.154551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.155684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] RBAC: Authorizing `identity:get_project(project_id=819299051284471fafff0f03b1919b16)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.157138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.158924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 523/1064] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/projects/819299051284471fafff0f03b1919b16 => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:42.164845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.187478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee871d22-96d4-43fa-b8ee-c730d4f12b74 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.218919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.222569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.223198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.223468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.223737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.231499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.232961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:42.244708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:42.245707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:42.246713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:42.249188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 542/1065] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:42.251294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.280621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.283601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1eb09d-9a3b-463c-ab68-c0896e488132 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.325499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.326139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.326420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.326693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.327412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=819299051284471fafff0f03b1919b16, user.id=9d403b95179b41c587682dbf53335624) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.327720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.329787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.334613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 524/1066] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/role_assignments?user.id=9d403b95179b41c587682dbf53335624&scope.project.id=819299051284471fafff0f03b1919b16&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:42.373753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0bab3e-b321-435c-be6e-b22192b07e47 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.411527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.412153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.412426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.412691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.413456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.413758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.415775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.439988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 543/1067] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:42.441120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.546885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226af1d7-3d03-461a-ae43-966bd486b609 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.560281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.583254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.583804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.584079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.584348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.595696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.597154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:42.611596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:42.612553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:42.613494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:42.613494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:42.613494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:42.613494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 525/1068] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:42.618349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.703388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.721110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90f796-7fe9-4972-8e78-7cf7af830dec None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.755089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.755699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.755982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.756257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.756915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.757224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.759124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.770408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 544/1069] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:42.776354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.877748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d222be-295a-4410-8778-37b7826acec9 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.910646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.911281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.911562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.911838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.919033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.920513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:42.931712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:42.932587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:42.933794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:42.934661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 526/1070] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:42.938645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.041528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94003e18-7c1b-4567-ad42-dac5fc70b97c None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:43.075714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:43.076322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.076597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.076890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.077574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:43.077877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:43.079303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc32f7-e98c-4521-bc2f-f4f00ceec302 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:43.095919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 545/1071] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:43.097582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.127218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:43.201882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb75db46-094c-4ab7-af40-a1c929904670 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:43.238316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe453c8e-ba89-41aa-99e1-d153c249cde1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:43.238980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe453c8e-ba89-41aa-99e1-d153c249cde1 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.239263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe453c8e-ba89-41aa-99e1-d153c249cde1 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.239538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe453c8e-ba89-41aa-99e1-d153c249cde1 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/e528956ee4704ac0b5ce4c62826b71ee/roles/c5cd5d31d6504254b5fa4fb776b12311` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.241774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe453c8e-ba89-41aa-99e1-d153c249cde1 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=e528956ee4704ac0b5ce4c62826b71ee, role_id=c5cd5d31d6504254b5fa4fb776b12311)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:43.243719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe453c8e-ba89-41aa-99e1-d153c249cde1 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:43.264964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 527/1072] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:43 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/e528956ee4704ac0b5ce4c62826b71ee/roles/c5cd5d31d6504254b5fa4fb776b12311 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:59:43.588575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eecb73-45e7-490f-b305-f8c3c02bc6ff None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.589019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eecb73-45e7-490f-b305-f8c3c02bc6ff None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.589192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eecb73-45e7-490f-b305-f8c3c02bc6ff None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.590632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 546/1073] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:43.595672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.596056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.596241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.614161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:43.615339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:43.687424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 528/1074] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:43.952918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eecb73-45e7-490f-b305-f8c3c02bc6ff None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.961670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.059520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eecb73-45e7-490f-b305-f8c3c02bc6ff None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.093118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.093724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.094001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.094271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.162287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.163737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.166980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.263504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 547/1075] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:44.269214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.286576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.371581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625c6aa1-b880-45b3-91b1-5a1a54c0ab25 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.406855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.407516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.407794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.408064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.419993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.421493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-49636dca-d215-4c44-a498-173597f6d60f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:44.436766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:44.437943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:44.438850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:44.438850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:44.438850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:44.438850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 529/1076] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:44.445233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.515449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98edef49-38c6-446b-9ca0-158bba9bdb29 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.515624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98edef49-38c6-446b-9ca0-158bba9bdb29 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.515898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98edef49-38c6-446b-9ca0-158bba9bdb29 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.517393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 530/1077] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:44.522193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.522497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.522800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.541778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:44.542971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:44.546103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8010d-4224-4426-a8bf-51ec6405cead None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.548974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.590058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.590744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.590945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.591221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.591844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:44.592148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.593532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.609035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 548/1078] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:44.618358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.624925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 531/1079] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:44.889973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.988623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eacee7-cdd9-48eb-9d1f-2724c2ae11ea None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.021761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.022389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.022665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.022935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.033975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:45.038679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.040261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:45.041275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:45.045074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 549/1080] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.046334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.141357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9055fab8-2230-413b-aea6-b52dc51d6a81 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.176014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.176617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.176922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.177193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.177854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.178153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.179500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.188479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 532/1081] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.194284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.218663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.291324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa99f767-90d0-44b4-9192-5ae950d9d4c1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.326852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c5be44-3469-4773-b50f-7e1142760770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.327501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5be44-3469-4773-b50f-7e1142760770 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.327780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5be44-3469-4773-b50f-7e1142760770 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.328050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5be44-3469-4773-b50f-7e1142760770 None admin] PATH_INFO: `/v3/users/9d403b95179b41c587682dbf53335624` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.329253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c5be44-3469-4773-b50f-7e1142760770 None admin] RBAC: Authorizing `identity:get_user(user_id=9d403b95179b41c587682dbf53335624)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.330719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c5be44-3469-4773-b50f-7e1142760770 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.332371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 550/1082] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/users/9d403b95179b41c587682dbf53335624 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.338402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.400668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.426788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f21d2-a111-41f5-978c-b3f6a57a0cb8 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.427053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f21d2-a111-41f5-978c-b3f6a57a0cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.427323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f21d2-a111-41f5-978c-b3f6a57a0cb8 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.428783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 551/1083] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:45.433724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.434010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.434290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.441864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9dadd-ff02-4ac3-b726-acee3f6b89cd None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.452230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:45.453443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:45.474982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.475600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.475940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.476223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.477406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] RBAC: Authorizing `identity:get_project(project_id=819299051284471fafff0f03b1919b16)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.478815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.480430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 533/1084] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/projects/819299051284471fafff0f03b1919b16 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.486192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.519517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 552/1085] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:45.580965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c241560-8745-4275-a6ff-05ab11947e29 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.608265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.616430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.617109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.617412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.617706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16/users/9d403b95179b41c587682dbf53335624/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.619561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] RBAC: Authorizing `identity:create_grant(project_id=819299051284471fafff0f03b1919b16, user_id=9d403b95179b41c587682dbf53335624, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.620997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.633256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 534/1086] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:45 2022] PUT /identity/v3/projects/819299051284471fafff0f03b1919b16/users/9d403b95179b41c587682dbf53335624/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:59:45.795110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.818682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.892447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48cb2a-c9b3-4d4d-bfe4-c351a18b71cb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.925282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.925912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.926195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.926465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.936362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:45.941189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.942415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:45.943856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 553/1087] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.949184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.046922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f832e-b709-4e72-bf33-7f40c7ee5b5c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.079489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.080086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.080364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.080633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.081327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.081627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.082973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.091340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 535/1088] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.097134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.175980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.194829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c3b657-c9b5-42e0-bfd0-97aec3da20d1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.227551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.228174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.228456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.228730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.238449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.239886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:46.254449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.255603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:46.257103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:46.257103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.257103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:46.257103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 554/1089] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.261772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.310366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.360826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d836295-b62f-4c1c-bf25-f227684dc0be None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.393639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.394241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.394519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.394789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.395420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.395720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.397094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.408756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 536/1090] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.414457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.514420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e04448d-0f6f-4b25-ae51-c425f6a98f61 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.546614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.547239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.547513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.547782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.549672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.556599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.558066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:46.569180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.570128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 01:59:46.578602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 555/1091] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.579250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.676432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548cd46-507a-4cfe-86a4-d84492884794 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.687409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.711246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.712351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.712630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.712924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.713637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.713939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.715337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.731764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 537/1092] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.737450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.834785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b290068-c186-4b07-b663-e83b4cd74728 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.868077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.868678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.868984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.869257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.926243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.934713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.936163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.030232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 556/1093] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:47.035943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.131838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d459753a-4d32-46ee-8964-e1e28862ec3d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.165280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.165898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.166176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.166447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.176088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.177555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:47.191560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:47.192967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:47.194015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:47.194015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:47.194015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:47.194015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 538/1094] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:47.198595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.290682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.297778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ed8dfb-6703-49bf-b1ba-b4a4e48b9327 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.331410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.332029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.332309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.332583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.333260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.333567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.334932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.346411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 557/1095] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.351990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.373422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.449404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c420cf-cc77-4c55-959c-67b098b99b80 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.481778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.482405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.482685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.482959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.489409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.490836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:47.501650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:47.502612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:47.503563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 539/1096] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:47.509492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.610272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee1be77-5a09-41cb-ace3-0114f3356aa2 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.646569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.647184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.647468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.647741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.648436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.648808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.650278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626a1525-6ef3-4837-9712-b1ffcf85edc4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.662458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 558/1097] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.668498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.736052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6792c8-7f50-4348-b12d-0b83dcef7216 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.736262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6792c8-7f50-4348-b12d-0b83dcef7216 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.736541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6792c8-7f50-4348-b12d-0b83dcef7216 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.737912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 559/1098] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:47.742988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.743271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.743542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.748556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.761658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:47.762832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:47.767719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b7db0f-540c-4807-a7e0-8b1c49e7a310 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.774478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.801157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.801755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.802033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.802302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.803009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] RBAC: Adding query filter params (role.id=c5cd5d31d6504254b5fa4fb776b12311, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=e528956ee4704ac0b5ce4c62826b71ee) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.803316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.804723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.810513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 540/1099] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/role_assignments?user.id=e528956ee4704ac0b5ce4c62826b71ee&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=c5cd5d31d6504254b5fa4fb776b12311 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.832851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 560/1100] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 01:59:48.102271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.200572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ca5136-1de4-40b1-a909-c3667575b235 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.234892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.235513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.235797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.236062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.247662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:48.252355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:48.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:48.254693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 01:59:48.257047 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 541/1101] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:48.259436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.359639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2c1575-cabb-4d39-a80b-f2f4a707a3cf None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.393746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.394359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.394640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.394910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.395577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:48.395880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.397330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.406049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 561/1102] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:48.412147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.415980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.517747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b313a5d-9562-4c15-9877-3382672a47ae None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.551451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.552080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.552359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.552634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] PATH_INFO: `/v3/users/9d403b95179b41c587682dbf53335624` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.553791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] RBAC: Authorizing `identity:get_user(user_id=9d403b95179b41c587682dbf53335624)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.555933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.557542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 542/1103] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/users/9d403b95179b41c587682dbf53335624 => generated 285 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:48.563392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.574364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.669197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad13f4e-d0b8-4d96-a5d3-e41c411b154e None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.702727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.703327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.703604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.703872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.768369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.769828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.777022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.863692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 562/1104] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:48.869653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.935680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.963826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54704ba-2eab-46a3-b863-66a3e3930dbc None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.999679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d28ba45-ac13-4c31-bec9-e76036791a98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.000681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d28ba45-ac13-4c31-bec9-e76036791a98 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.001232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d28ba45-ac13-4c31-bec9-e76036791a98 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.001622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d28ba45-ac13-4c31-bec9-e76036791a98 None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.002731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d28ba45-ac13-4c31-bec9-e76036791a98 None admin] RBAC: Authorizing `identity:get_project(project_id=819299051284471fafff0f03b1919b16)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.004209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d28ba45-ac13-4c31-bec9-e76036791a98 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.005901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 543/1105] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/projects/819299051284471fafff0f03b1919b16 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:49.012018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.105679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32115dab-7e3a-4fba-8f7a-a1f8e1e76330 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.112531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.138681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.139265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.139541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.139810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.140510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=819299051284471fafff0f03b1919b16, user.id=9d403b95179b41c587682dbf53335624) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:49.140811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.142254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a72108-c776-4462-bcaf-c7fc1c10fe96 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.147734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 563/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:49 2022] GET /identity/v3/role_assignments?user.id=9d403b95179b41c587682dbf53335624&scope.project.id=819299051284471fafff0f03b1919b16&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:51.284398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4ef35a-0707-4909-8b43-adfdf90da06a None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.284944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4ef35a-0707-4909-8b43-adfdf90da06a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.285063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4ef35a-0707-4909-8b43-adfdf90da06a None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.286306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 544/1107] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:51.292104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.292481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.292660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.310985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:51.312171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:51.376833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 564/1108] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:51.646852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4ef35a-0707-4909-8b43-adfdf90da06a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.744563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4ef35a-0707-4909-8b43-adfdf90da06a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.776806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0ed1cb-4f54-489f-8312-c7d8b9090d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.777437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ed1cb-4f54-489f-8312-c7d8b9090d40 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.777720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ed1cb-4f54-489f-8312-c7d8b9090d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.777993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ed1cb-4f54-489f-8312-c7d8b9090d40 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.779100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0ed1cb-4f54-489f-8312-c7d8b9090d40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.780512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0ed1cb-4f54-489f-8312-c7d8b9090d40 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.782075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 545/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:51.787576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.877434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.880270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56121e0-38b7-456c-a4cf-703233878ae3 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.913414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1507bd63-5b1f-43a8-8377-a209bc52050f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.913999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1507bd63-5b1f-43a8-8377-a209bc52050f None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.914274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1507bd63-5b1f-43a8-8377-a209bc52050f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.914539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1507bd63-5b1f-43a8-8377-a209bc52050f None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.915167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1507bd63-5b1f-43a8-8377-a209bc52050f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.916573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1507bd63-5b1f-43a8-8377-a209bc52050f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.936682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 565/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:51 2022] POST /identity/v3/users => generated 286 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:54.133957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92bb011-69b3-4e6a-85b4-34e3740bcfdd None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.134485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92bb011-69b3-4e6a-85b4-34e3740bcfdd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.134485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92bb011-69b3-4e6a-85b4-34e3740bcfdd None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.135820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 546/1111] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:54.141066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.141445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.141615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.159894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:54.161112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:54.225252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 566/1112] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:54.493120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92bb011-69b3-4e6a-85b4-34e3740bcfdd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.504054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.591635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92bb011-69b3-4e6a-85b4-34e3740bcfdd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.623832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.624449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.624729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.625031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] PATH_INFO: `/v3/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.626399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] RBAC: Authorizing `identity:get_role(role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.627743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.629359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 547/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/roles/609c7d98c34d4dba86ff458ea0726951 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.635725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.716124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.731624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc905fe8-604f-4c8b-a50b-e538d883eb65 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.764708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.765357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.765638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.765909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] PATH_INFO: `/v3/users/6be78f6683394c67a8cf77224d4c6bd3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.778393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] RBAC: Authorizing `identity:get_user(user_id=6be78f6683394c67a8cf77224d4c6bd3)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.779825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.781444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 567/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/users/6be78f6683394c67a8cf77224d4c6bd3 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.787108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.861243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.883774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9f4676-77f6-4e92-9077-95871f6001a7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.915644 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.916233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.916510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.916778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.979916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.981383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.052003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.075203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 548/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 01:59:55.081002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.175745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5701394e-f606-446a-957d-a21097e16fe5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.207671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b49b59-9840-4d1c-a01f-8c05f7a7d213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.208287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b49b59-9840-4d1c-a01f-8c05f7a7d213 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.208565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b49b59-9840-4d1c-a01f-8c05f7a7d213 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.208837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b49b59-9840-4d1c-a01f-8c05f7a7d213 None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.209979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b49b59-9840-4d1c-a01f-8c05f7a7d213 None admin] RBAC: Authorizing `identity:get_project(project_id=819299051284471fafff0f03b1919b16)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.211901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b49b59-9840-4d1c-a01f-8c05f7a7d213 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.213543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 568/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/projects/819299051284471fafff0f03b1919b16 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:55.219666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.312688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dc0f10-87ea-40f0-83cd-0dfee12f4ab6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.345154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.345742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.346022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.346291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.347003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] RBAC: Adding query filter params (role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=819299051284471fafff0f03b1919b16, user.id=6be78f6683394c67a8cf77224d4c6bd3) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:55.347311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.348716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284b924-8cf2-4568-83b9-4201ef260df4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.354478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 549/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/role_assignments?user.id=6be78f6683394c67a8cf77224d4c6bd3&scope.project.id=819299051284471fafff0f03b1919b16&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 259 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:57.440105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838affb5-4776-4f6a-9037-7622ca29949d None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.440634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838affb5-4776-4f6a-9037-7622ca29949d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.440634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838affb5-4776-4f6a-9037-7622ca29949d None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.441999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 569/1118] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 01:59:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:57.447826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.448191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.448366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.465940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:57.467109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:57.499393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.531440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 550/1119] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 01:59:57.800216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838affb5-4776-4f6a-9037-7622ca29949d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:57.815283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.898486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838affb5-4776-4f6a-9037-7622ca29949d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.931215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.931824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.932099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.932367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] PATH_INFO: `/v3/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.933745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] RBAC: Authorizing `identity:get_role(role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.935093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.936373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 570/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/roles/609c7d98c34d4dba86ff458ea0726951 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:57.942137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.036012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13266b-fe23-474d-9061-cc000681ec7f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.069525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.070084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.070396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.070669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] PATH_INFO: `/v3/users/6be78f6683394c67a8cf77224d4c6bd3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.071780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] RBAC: Authorizing `identity:get_user(user_id=6be78f6683394c67a8cf77224d4c6bd3)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.073228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.074760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 551/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/users/6be78f6683394c67a8cf77224d4c6bd3 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:58.080200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.158700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.177507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7af92e8-b38a-4726-9e4b-e7e1ff0ec8bc None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.209506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d52081-e2cd-4438-9289-8d84a36b7d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.210118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d52081-e2cd-4438-9289-8d84a36b7d9d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.210393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d52081-e2cd-4438-9289-8d84a36b7d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.210659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d52081-e2cd-4438-9289-8d84a36b7d9d None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.212235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d52081-e2cd-4438-9289-8d84a36b7d9d None admin] RBAC: Authorizing `identity:get_project(project_id=819299051284471fafff0f03b1919b16)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.213670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d52081-e2cd-4438-9289-8d84a36b7d9d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.215288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 571/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/projects/819299051284471fafff0f03b1919b16 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:58.221248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.259354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.316025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c952558-a271-4f37-b0c8-7cd30a3cc899 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.348523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa313600-f9f4-42a6-8e62-4d6221f3f4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.349146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa313600-f9f4-42a6-8e62-4d6221f3f4f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.349428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa313600-f9f4-42a6-8e62-4d6221f3f4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.349699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa313600-f9f4-42a6-8e62-4d6221f3f4f0 None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16/users/6be78f6683394c67a8cf77224d4c6bd3/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.352348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa313600-f9f4-42a6-8e62-4d6221f3f4f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=819299051284471fafff0f03b1919b16, user_id=6be78f6683394c67a8cf77224d4c6bd3, role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.353796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa313600-f9f4-42a6-8e62-4d6221f3f4f0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.365616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 552/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:58 2022] PUT /identity/v3/projects/819299051284471fafff0f03b1919b16/users/6be78f6683394c67a8cf77224d4c6bd3/roles/609c7d98c34d4dba86ff458ea0726951 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:00.445748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e805a039-f9d9-4363-8eec-2dd6c8b25bfb None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.446308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e805a039-f9d9-4363-8eec-2dd6c8b25bfb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.446308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e805a039-f9d9-4363-8eec-2dd6c8b25bfb None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.447603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 572/1124] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:00.452702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.453152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.453262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.471272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:00.472446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:00.535400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 553/1125] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:00.802385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e805a039-f9d9-4363-8eec-2dd6c8b25bfb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:00.898379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e805a039-f9d9-4363-8eec-2dd6c8b25bfb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:00.930417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:00.931026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.931298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.931565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] PATH_INFO: `/v3/roles/609c7d98c34d4dba86ff458ea0726951` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.932940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] RBAC: Authorizing `identity:get_role(role_id=609c7d98c34d4dba86ff458ea0726951)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:00.934289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:00.935569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 573/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:00 2022] GET /identity/v3/roles/609c7d98c34d4dba86ff458ea0726951 => generated 223 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:00.941195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:00.968902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:01.042968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0802eb-51e0-4dd4-b196-a7662e804838 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:01.076037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:01.076598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:01.076897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:01.077183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] PATH_INFO: `/v3/users/6be78f6683394c67a8cf77224d4c6bd3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:01.078299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] RBAC: Authorizing `identity:get_user(user_id=6be78f6683394c67a8cf77224d4c6bd3)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:01.079718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:01.081328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 554/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:00 2022] GET /identity/v3/users/6be78f6683394c67a8cf77224d4c6bd3 => generated 286 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:01.087568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:01.104747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:01.182507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44ebb46-0893-4eda-a6b6-8587b2f96931 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:01.215532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:01.216121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:01.216395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:01.216660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:01.279493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:01.280931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:01.296898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:01.375471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 574/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:01.381299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:01.469742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:01.475684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dbab3f-6b77-4c7a-9ead-4f19af38c3ac None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:01.507551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0e8ca9-3c11-4a83-a757-4faabc14ce7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:01.508725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0e8ca9-3c11-4a83-a757-4faabc14ce7d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:01.509027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0e8ca9-3c11-4a83-a757-4faabc14ce7d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:01.509296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0e8ca9-3c11-4a83-a757-4faabc14ce7d None admin] PATH_INFO: `/v3/projects/819299051284471fafff0f03b1919b16` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:01.510384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0e8ca9-3c11-4a83-a757-4faabc14ce7d None admin] RBAC: Authorizing `identity:get_project(project_id=819299051284471fafff0f03b1919b16)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:01.511780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0e8ca9-3c11-4a83-a757-4faabc14ce7d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:01.513393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 555/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:01 2022] GET /identity/v3/projects/819299051284471fafff0f03b1919b16 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:01.519324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:01.614623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a86dce-c8fb-410a-a7ce-7a2df282ad61 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:01.630127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:01.647900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:01.649069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:01.649350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:01.649621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:01.650328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] RBAC: Adding query filter params (role.id=609c7d98c34d4dba86ff458ea0726951, scope.project.id=819299051284471fafff0f03b1919b16, user.id=6be78f6683394c67a8cf77224d4c6bd3) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:01.650632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:01.652086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246db488-ad84-464d-b671-c051f035cce4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:01.657653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 575/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:01 2022] GET /identity/v3/role_assignments?user.id=6be78f6683394c67a8cf77224d4c6bd3&scope.project.id=819299051284471fafff0f03b1919b16&role.id=609c7d98c34d4dba86ff458ea0726951 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:03.793274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab47e3a1-f77f-4d2b-a191-eea72206cbe4 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.793792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab47e3a1-f77f-4d2b-a191-eea72206cbe4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.793886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab47e3a1-f77f-4d2b-a191-eea72206cbe4 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.795145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 556/1131] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:03.800348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.800745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.800922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.819421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:03.820651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:03.885257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 576/1132] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:04.160220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab47e3a1-f77f-4d2b-a191-eea72206cbe4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.258505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab47e3a1-f77f-4d2b-a191-eea72206cbe4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.291404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd5ee8b-e9be-4e67-a445-6cfc6cdfe677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.292016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd5ee8b-e9be-4e67-a445-6cfc6cdfe677 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.292299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd5ee8b-e9be-4e67-a445-6cfc6cdfe677 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.292573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd5ee8b-e9be-4e67-a445-6cfc6cdfe677 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.293713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd5ee8b-e9be-4e67-a445-6cfc6cdfe677 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.295144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd5ee8b-e9be-4e67-a445-6cfc6cdfe677 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.296693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 557/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:04.302511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.397559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6d4ca-6bf7-4433-9366-6198fc846bd3 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.408245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.431790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9900cc4-6d42-495b-aeee-7702f83ea427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.432328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9900cc4-6d42-495b-aeee-7702f83ea427 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.432608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9900cc4-6d42-495b-aeee-7702f83ea427 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.432905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9900cc4-6d42-495b-aeee-7702f83ea427 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.433545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9900cc4-6d42-495b-aeee-7702f83ea427 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.434971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9900cc4-6d42-495b-aeee-7702f83ea427 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.460096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 577/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:04 2022] POST /identity/v3/projects => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:06.596094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43356ce6-6bd7-47df-a950-679de92c1e69 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.596523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43356ce6-6bd7-47df-a950-679de92c1e69 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.596646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43356ce6-6bd7-47df-a950-679de92c1e69 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.598047 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 558/1135] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:06.603303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.603689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.603850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.621411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:06.622569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:06.686589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 578/1136] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:06.952771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43356ce6-6bd7-47df-a950-679de92c1e69 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.982285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:07.048911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43356ce6-6bd7-47df-a950-679de92c1e69 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.081535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8040a0fc-400f-4387-9ce3-8bd65cbd9be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.082147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8040a0fc-400f-4387-9ce3-8bd65cbd9be0 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.082450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8040a0fc-400f-4387-9ce3-8bd65cbd9be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.082719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8040a0fc-400f-4387-9ce3-8bd65cbd9be0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.083807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8040a0fc-400f-4387-9ce3-8bd65cbd9be0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.085254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8040a0fc-400f-4387-9ce3-8bd65cbd9be0 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.086802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 559/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:07.092502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:07.176542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:07.186298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be05b7e-2531-41c7-9007-6389d14c6825 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.219553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4e3732-1664-4daf-b58a-143979ff24b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.220083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4e3732-1664-4daf-b58a-143979ff24b0 None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.220357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4e3732-1664-4daf-b58a-143979ff24b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.220623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4e3732-1664-4daf-b58a-143979ff24b0 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.221294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4e3732-1664-4daf-b58a-143979ff24b0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.222704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4e3732-1664-4daf-b58a-143979ff24b0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.244109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 579/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 02:00:07 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:09.405785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ac59c-2fa1-4ea4-b8e6-95e44fdca0f8 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.406338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ac59c-2fa1-4ea4-b8e6-95e44fdca0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.406338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ac59c-2fa1-4ea4-b8e6-95e44fdca0f8 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.407649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 560/1139] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:09.412779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.413206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.413344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.431646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:09.432812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:09.495610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 580/1140] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:09.760629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24ac59c-2fa1-4ea4-b8e6-95e44fdca0f8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:09.854041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:09.858003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24ac59c-2fa1-4ea4-b8e6-95e44fdca0f8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:09.890553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:09.891173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.891454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.891726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.901930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:09.906660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:09.907791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:00:09.908812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 561/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:09.913603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:09.987929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.008666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f42265-9d17-49e1-8b41-dc2185e56e9f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.040798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.041360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.041638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.041906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.042570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:10.042868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.044224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.051819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 581/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:10.058281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.154218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce360c9d-777b-4a65-a311-1e12198c0676 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.188746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.189430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.189712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.190006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] PATH_INFO: `/v3/users/51b10b1d528147baab2267ba78b6c8d1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.203236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] RBAC: Authorizing `identity:get_user(user_id=51b10b1d528147baab2267ba78b6c8d1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.204659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.206290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 562/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/users/51b10b1d528147baab2267ba78b6c8d1 => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:10.212527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.308240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3c7e12-4efa-4a5f-a69a-1c1b89862886 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.341378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.341970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.342243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.342508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.352441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.407504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.408965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.501279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 582/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:10.507328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.526494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.606928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821b6a37-2c52-4b88-b9bc-23da18690c78 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.638705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96b86f6-b9a1-4678-a654-a4d908a2d574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.639313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96b86f6-b9a1-4678-a654-a4d908a2d574 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.639587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96b86f6-b9a1-4678-a654-a4d908a2d574 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.639854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96b86f6-b9a1-4678-a654-a4d908a2d574 None admin] PATH_INFO: `/v3/projects/74daf38831264efbaf9d0e01866082b0` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.640965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96b86f6-b9a1-4678-a654-a4d908a2d574 None admin] RBAC: Authorizing `identity:get_project(project_id=74daf38831264efbaf9d0e01866082b0)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.642372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96b86f6-b9a1-4678-a654-a4d908a2d574 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.643948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 563/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/projects/74daf38831264efbaf9d0e01866082b0 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:10.650630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.690526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.746081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491701a3-7aa4-42d5-bf8f-3dbd70575371 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.778193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.778781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.779058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.779328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.780033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=74daf38831264efbaf9d0e01866082b0, user.id=51b10b1d528147baab2267ba78b6c8d1) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:10.780338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.781782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b02509-bf1f-48c3-844f-102a958c93e2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.786893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 583/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/role_assignments?user.id=51b10b1d528147baab2267ba78b6c8d1&scope.project.id=74daf38831264efbaf9d0e01866082b0&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:12.896071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e824f232-bd3c-4a75-9bd0-da08ae831af9 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.896571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e824f232-bd3c-4a75-9bd0-da08ae831af9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.896571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e824f232-bd3c-4a75-9bd0-da08ae831af9 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.898001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 564/1147] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:12.903116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.903550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.903690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.921169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:12.922338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:12.985180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 584/1148] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:13.259374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e824f232-bd3c-4a75-9bd0-da08ae831af9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.341346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.357439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e824f232-bd3c-4a75-9bd0-da08ae831af9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.390435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.391040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.391316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.391583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.401211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:13.405832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:13.406939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:00:13.408219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 565/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:13.412885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.445826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.509239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d500625f-1d7d-48d0-9c53-2dddb73db13f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.541463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.541999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.542276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.542547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.543195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:13.543494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.544846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.552461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 585/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.558804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.653660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b995fd2-473b-48f7-895d-d5471227b597 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.685434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.686042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.686316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.686582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] PATH_INFO: `/v3/users/51b10b1d528147baab2267ba78b6c8d1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.687675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] RBAC: Authorizing `identity:get_user(user_id=51b10b1d528147baab2267ba78b6c8d1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.689130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.690688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 566/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/users/51b10b1d528147baab2267ba78b6c8d1 => generated 286 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.696680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.786020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.792468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d389bd-65c4-448e-a4e1-07e59d2e2044 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.824049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aa4556-0f74-4351-a342-86800f0724d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.824675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aa4556-0f74-4351-a342-86800f0724d7 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.824965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aa4556-0f74-4351-a342-86800f0724d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.825238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aa4556-0f74-4351-a342-86800f0724d7 None admin] PATH_INFO: `/v3/projects/74daf38831264efbaf9d0e01866082b0` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.826335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aa4556-0f74-4351-a342-86800f0724d7 None admin] RBAC: Authorizing `identity:get_project(project_id=74daf38831264efbaf9d0e01866082b0)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.827754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aa4556-0f74-4351-a342-86800f0724d7 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.829373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 586/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/projects/74daf38831264efbaf9d0e01866082b0 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.835104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.850795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.929838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465af84-37ed-43e4-9a35-b2743e312d8b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.962030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb02856-eb57-4e99-a5d5-d33964f01db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.962662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb02856-eb57-4e99-a5d5-d33964f01db2 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.962937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb02856-eb57-4e99-a5d5-d33964f01db2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.963204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb02856-eb57-4e99-a5d5-d33964f01db2 None admin] PATH_INFO: `/v3/projects/74daf38831264efbaf9d0e01866082b0/users/51b10b1d528147baab2267ba78b6c8d1/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.965068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb02856-eb57-4e99-a5d5-d33964f01db2 None admin] RBAC: Authorizing `identity:create_grant(project_id=74daf38831264efbaf9d0e01866082b0, user_id=51b10b1d528147baab2267ba78b6c8d1, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.966502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb02856-eb57-4e99-a5d5-d33964f01db2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.979134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 567/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:13 2022] PUT /identity/v3/projects/74daf38831264efbaf9d0e01866082b0/users/51b10b1d528147baab2267ba78b6c8d1/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:00:16.076136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18477fb-529e-48a0-bfd5-f1951770f408 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.076771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18477fb-529e-48a0-bfd5-f1951770f408 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.076771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18477fb-529e-48a0-bfd5-f1951770f408 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.078283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 587/1154] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:16.083505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.083888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.084056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.101732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:16.102917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:16.167370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 568/1155] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:16.444256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18477fb-529e-48a0-bfd5-f1951770f408 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.541793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18477fb-529e-48a0-bfd5-f1951770f408 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.574387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.575005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.575282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.575553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.586376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:16.591100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:16.592283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:00:16.593418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 588/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:16.598131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.601141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.698256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227d1817-37ea-4d23-8a66-c0c76ac92ead None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.729748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.730342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.730617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.730883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.731530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:16.731833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:16.733198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:16.740727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 569/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:16.747184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.772165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.842941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b13062-b2e8-4957-832e-305378eb9bb4 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.874777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.875386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.875663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.875931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] PATH_INFO: `/v3/users/51b10b1d528147baab2267ba78b6c8d1` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.877051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] RBAC: Authorizing `identity:get_user(user_id=51b10b1d528147baab2267ba78b6c8d1)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:16.878466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:16.880013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 589/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/users/51b10b1d528147baab2267ba78b6c8d1 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:16.886226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.948943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.982177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf3fdc-5f9e-42dc-9dc2-f9c406d8eb91 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.014410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.014999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.015280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.015552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.079114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.080551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.140496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.174727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 570/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:17.181091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.268437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.278014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0369d-e012-4d6d-9d13-af2817e4120a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.309286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d58372-d15c-40a1-95c9-6beda9cf4968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.309899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d58372-d15c-40a1-95c9-6beda9cf4968 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.310178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d58372-d15c-40a1-95c9-6beda9cf4968 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.310448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d58372-d15c-40a1-95c9-6beda9cf4968 None admin] PATH_INFO: `/v3/projects/74daf38831264efbaf9d0e01866082b0` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.311540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d58372-d15c-40a1-95c9-6beda9cf4968 None admin] RBAC: Authorizing `identity:get_project(project_id=74daf38831264efbaf9d0e01866082b0)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.312990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d58372-d15c-40a1-95c9-6beda9cf4968 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.314583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 590/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/projects/74daf38831264efbaf9d0e01866082b0 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.321375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.415871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42258623-70a3-4179-9095-a7591eb03d0b None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.449896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.450483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.450761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.451033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.451751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=74daf38831264efbaf9d0e01866082b0, user.id=51b10b1d528147baab2267ba78b6c8d1) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.452059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.453507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d30164d-992f-4985-bc4d-1d55e79d0710 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.459079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 571/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/role_assignments?user.id=51b10b1d528147baab2267ba78b6c8d1&scope.project.id=74daf38831264efbaf9d0e01866082b0&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:19.573493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fef2f7-f074-405a-9db6-59a531e410e2 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.574013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fef2f7-f074-405a-9db6-59a531e410e2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.574130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78fef2f7-f074-405a-9db6-59a531e410e2 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.575391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 591/1162] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:19.580555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.581021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.581121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.598752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:19.599918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:19.622998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:19.663376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 572/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:19.931391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fef2f7-f074-405a-9db6-59a531e410e2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:20.028186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78fef2f7-f074-405a-9db6-59a531e410e2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:20.063324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:20.063923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:20.064403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:20.064934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:20.092081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:20.130726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:20.132167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:20.224124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 592/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:20.230630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:20.325715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4190fd14-d343-4a1d-9b47-725fa680afd9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:20.357990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:20.358603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:20.358882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:20.359150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:20.365497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:20.366961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-a46ebada-3a71-4160-b83a-a05d2265e848 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:20.377872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:20.379050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:00:20.380108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 573/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:20 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:20.385595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:20.443156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:20.482499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e163f4d-1b73-4dd8-a182-52addffe9ae1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:20.515588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:20.516185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:20.516465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:20.516738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:20.517400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:20.517702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:20.519064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1762ee81-c69d-4274-a704-c64ee462b1df None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:20.525893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 593/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:20 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:22.647890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5db98ad-930c-4790-8011-74e5b2f37547 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.648941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5db98ad-930c-4790-8011-74e5b2f37547 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.649052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5db98ad-930c-4790-8011-74e5b2f37547 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.650341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 574/1167] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:22.655612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e2767-afd2-4ac5-90b7-16aaac7ad9d8 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.655992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e2767-afd2-4ac5-90b7-16aaac7ad9d8 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.656167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e2767-afd2-4ac5-90b7-16aaac7ad9d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.673853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-255e2767-afd2-4ac5-90b7-16aaac7ad9d8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:22.675032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-255e2767-afd2-4ac5-90b7-16aaac7ad9d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:22.744690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 594/1168] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:23.009566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5db98ad-930c-4790-8011-74e5b2f37547 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.031506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.109864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5db98ad-930c-4790-8011-74e5b2f37547 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.147508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67103704-b85a-48ac-b798-510d15603690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.148680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67103704-b85a-48ac-b798-510d15603690 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.148984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67103704-b85a-48ac-b798-510d15603690 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.149257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67103704-b85a-48ac-b798-510d15603690 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.149928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67103704-b85a-48ac-b798-510d15603690 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.151281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67103704-b85a-48ac-b798-510d15603690 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.166348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 575/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Thu Sep 29 02:00:23 2022] POST /identity/v3/domains => generated 267 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:25.327434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c528201-4a07-4a41-84a7-8d8414b2c62c None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.327979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c528201-4a07-4a41-84a7-8d8414b2c62c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.327979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c528201-4a07-4a41-84a7-8d8414b2c62c None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.329336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 595/1170] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:25.334572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.334959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.335163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.353545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:25.354732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:25.418277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 576/1171] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:25.685656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c528201-4a07-4a41-84a7-8d8414b2c62c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:25.740026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:25.784215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c528201-4a07-4a41-84a7-8d8414b2c62c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:25.816433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae81347-c1e6-42bf-ae27-bbb460b9657a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:25.817073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae81347-c1e6-42bf-ae27-bbb460b9657a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.817363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae81347-c1e6-42bf-ae27-bbb460b9657a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.817895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae81347-c1e6-42bf-ae27-bbb460b9657a None admin] PATH_INFO: `/v3/domains/100ce5bcbab14c209d9f4c484407db48` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.828363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae81347-c1e6-42bf-ae27-bbb460b9657a None admin] RBAC: Authorizing `identity:get_domain(domain_id=100ce5bcbab14c209d9f4c484407db48)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:25.829832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae81347-c1e6-42bf-ae27-bbb460b9657a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:25.831396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 596/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Sep 29 02:00:25 2022] GET /identity/v3/domains/100ce5bcbab14c209d9f4c484407db48 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:25.837393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:25.848415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:25.931879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6cf8ca-f362-452f-a284-7a880441a945 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:25.964176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9c1d27-e84d-4783-8c0b-ca131d377cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:25.964766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9c1d27-e84d-4783-8c0b-ca131d377cca None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.965072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9c1d27-e84d-4783-8c0b-ca131d377cca None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.965345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9c1d27-e84d-4783-8c0b-ca131d377cca None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.965980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9c1d27-e84d-4783-8c0b-ca131d377cca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:25.967390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9c1d27-e84d-4783-8c0b-ca131d377cca None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:25.992370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 577/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:25 2022] POST /identity/v3/projects => generated 361 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:28.143971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6fd268-4d9f-4908-a09a-5d87912318ce None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.144536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6fd268-4d9f-4908-a09a-5d87912318ce None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.144536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6fd268-4d9f-4908-a09a-5d87912318ce None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.145877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 597/1174] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:28.151607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.152041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.152207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.170549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:28.171728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:28.224703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.236249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 578/1175] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:28.503623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6fd268-4d9f-4908-a09a-5d87912318ce None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.605878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6fd268-4d9f-4908-a09a-5d87912318ce None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.619164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.639032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b28e63-a304-4f69-a346-ad35e0ae267a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.639660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b28e63-a304-4f69-a346-ad35e0ae267a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.639938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b28e63-a304-4f69-a346-ad35e0ae267a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.640207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b28e63-a304-4f69-a346-ad35e0ae267a None admin] PATH_INFO: `/v3/domains/100ce5bcbab14c209d9f4c484407db48` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.641325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b28e63-a304-4f69-a346-ad35e0ae267a None admin] RBAC: Authorizing `identity:get_domain(domain_id=100ce5bcbab14c209d9f4c484407db48)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.642753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b28e63-a304-4f69-a346-ad35e0ae267a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.644305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 598/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/domains/100ce5bcbab14c209d9f4c484407db48 => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:28.650649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.746601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b29d34-e206-4e05-b5e2-4c4cd102a929 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.782481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb99193d-a0c4-40b2-ac30-40c2e37eff59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.783007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb99193d-a0c4-40b2-ac30-40c2e37eff59 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.783286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb99193d-a0c4-40b2-ac30-40c2e37eff59 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.783557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb99193d-a0c4-40b2-ac30-40c2e37eff59 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.784192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb99193d-a0c4-40b2-ac30-40c2e37eff59 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.785649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb99193d-a0c4-40b2-ac30-40c2e37eff59 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.806999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 579/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 02:00:28 2022] POST /identity/v3/users => generated 311 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:30.961858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a6076-d9de-44ff-b87a-bec6dab12c33 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:30.962369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a6076-d9de-44ff-b87a-bec6dab12c33 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:30.962369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a6076-d9de-44ff-b87a-bec6dab12c33 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:30.963636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 599/1178] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:30.968862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:30.969308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:30.969401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:30.986865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:30.988032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:31.029323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:31.052244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 580/1179] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:31.327570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a6076-d9de-44ff-b87a-bec6dab12c33 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:31.423302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a6076-d9de-44ff-b87a-bec6dab12c33 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:31.458237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:31.458894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.459166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.459431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.470956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:31.478111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:31.479295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:00:31.480328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 600/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:31.485309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:31.578793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807949b1-e26c-41db-827a-ba6d78f29bc8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:31.611241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:31.611839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.612116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.612384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.613062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:31.613359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:31.614704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:31.622802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 581/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:31.628688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:31.670779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:31.725962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db335f-dc4c-487d-8224-5f26ae32c552 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:31.759730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:31.760334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.760607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.760896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] PATH_INFO: `/v3/users/53d0274239d44436946eef68ccbc986c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.773801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] RBAC: Authorizing `identity:get_user(user_id=53d0274239d44436946eef68ccbc986c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:31.775228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:31.776794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 601/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:31 2022] GET /identity/v3/users/53d0274239d44436946eef68ccbc986c => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:31.783451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:31.823926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:31.879656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba95ebba-e685-443c-93b8-6b2a00f2c6ee None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:31.910990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:31.911577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.911852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.912119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.975521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:31.976991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.015238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.071674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 582/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:32.077525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.169822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.172639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f7a5f-5957-4a7d-b018-134d3ab38c2a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.205811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff026f9-fc07-42e8-86ea-48354db62db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.206460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff026f9-fc07-42e8-86ea-48354db62db9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.206741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff026f9-fc07-42e8-86ea-48354db62db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.207014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff026f9-fc07-42e8-86ea-48354db62db9 None admin] PATH_INFO: `/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.208113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff026f9-fc07-42e8-86ea-48354db62db9 None admin] RBAC: Authorizing `identity:get_project(project_id=7ca9ceea147b4c7ead1cab8c9dcefc38)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.209565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff026f9-fc07-42e8-86ea-48354db62db9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.211254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 602/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:32.217403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.311381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ff8bb-f810-47ea-8ce0-d7625edbbeb5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.343200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.343791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.344071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.344343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.345075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=7ca9ceea147b4c7ead1cab8c9dcefc38, user.id=53d0274239d44436946eef68ccbc986c) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:32.345383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.346784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2054031-a293-4c8b-812b-05402df2216d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.352384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 583/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/role_assignments?user.id=53d0274239d44436946eef68ccbc986c&scope.project.id=7ca9ceea147b4c7ead1cab8c9dcefc38&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:34.450855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426b191-b5fd-40b7-a5b1-baa128f6c8fc None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.451384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426b191-b5fd-40b7-a5b1-baa128f6c8fc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.451384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426b191-b5fd-40b7-a5b1-baa128f6c8fc None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.452709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 603/1186] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:34.457913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.458350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.458444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.465722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:34.477934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:34.479113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:34.541871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 584/1187] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:34.814381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426b191-b5fd-40b7-a5b1-baa128f6c8fc None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:34.909835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426b191-b5fd-40b7-a5b1-baa128f6c8fc None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:34.941400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:34.942007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.942280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.942546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.952815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:34.957676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:34.958601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:00:34.959449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 604/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:34.964646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.059285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2597f53e-abfa-4452-a2b4-ad07a8f2d4f4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.091226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.091764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.092045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.092318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.092999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:35.093303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.094647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.095939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.103640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 585/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:35.109383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.138492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.207676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2eac4-a119-4dc9-8be8-e989131c4d86 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.239787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.240399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.240674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.241043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] PATH_INFO: `/v3/users/53d0274239d44436946eef68ccbc986c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.242699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] RBAC: Authorizing `identity:get_user(user_id=53d0274239d44436946eef68ccbc986c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.244109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.245677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 605/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/users/53d0274239d44436946eef68ccbc986c => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:35.251619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.345699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ca40d9-f28c-4044-b59e-ac46151aad01 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.379281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce0ca7-4b4a-438b-84c2-68da616f93cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.380090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce0ca7-4b4a-438b-84c2-68da616f93cf None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.380446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce0ca7-4b4a-438b-84c2-68da616f93cf None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.380719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce0ca7-4b4a-438b-84c2-68da616f93cf None admin] PATH_INFO: `/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.382057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce0ca7-4b4a-438b-84c2-68da616f93cf None admin] RBAC: Authorizing `identity:get_project(project_id=7ca9ceea147b4c7ead1cab8c9dcefc38)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.383712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce0ca7-4b4a-438b-84c2-68da616f93cf None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.385412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 586/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:35.391352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.485215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.491370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2550903f-7658-42ad-b4cd-ec608f2c2fa4 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.524028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380e8267-aef4-4839-98f6-1aa556f46168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.524673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e8267-aef4-4839-98f6-1aa556f46168 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.524982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e8267-aef4-4839-98f6-1aa556f46168 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.525256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e8267-aef4-4839-98f6-1aa556f46168 None admin] PATH_INFO: `/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38/users/53d0274239d44436946eef68ccbc986c/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.527134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e8267-aef4-4839-98f6-1aa556f46168 None admin] RBAC: Authorizing `identity:create_grant(project_id=7ca9ceea147b4c7ead1cab8c9dcefc38, user_id=53d0274239d44436946eef68ccbc986c, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.528574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e8267-aef4-4839-98f6-1aa556f46168 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.541208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 606/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:35 2022] PUT /identity/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38/users/53d0274239d44436946eef68ccbc986c/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:37.673826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af75ac3-d2ac-4dcc-85e1-3790c34a07de None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:37.674326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af75ac3-d2ac-4dcc-85e1-3790c34a07de None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:37.674415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af75ac3-d2ac-4dcc-85e1-3790c34a07de None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:37.675699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 587/1193] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:00:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:37.681505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:37.681883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:37.682046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:37.700485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:37.701687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:37.767285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 607/1194] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:38.035726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af75ac3-d2ac-4dcc-85e1-3790c34a07de None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.097391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:38.138615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af75ac3-d2ac-4dcc-85e1-3790c34a07de None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:38.176579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:38.177218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.177496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.177765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.187676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:38.193064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:38.194391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:00:38.196154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 588/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:38.200921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.280384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:38.295604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1951d18c-c8ac-45ba-b238-c290e0b3aaf8 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:38.327877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:38.328463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.328739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.329037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.329686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:38.329981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:38.331324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:38.338989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 608/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:38.344786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.439766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd87dff-9862-4403-b609-96e95a756f16 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:38.462555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:38.473485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:38.474104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.474388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.474663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] PATH_INFO: `/v3/users/53d0274239d44436946eef68ccbc986c` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.475773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] RBAC: Authorizing `identity:get_user(user_id=53d0274239d44436946eef68ccbc986c)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:38.477248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:38.478848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 589/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/users/53d0274239d44436946eef68ccbc986c => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:38.484624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.581352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33da1d2e-7496-4b94-8972-bb1c32782dbb None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:38.612989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:38.613586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.613865 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.614136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.630990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:38.677988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:38.679958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:38.772254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 609/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:38.778681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.880909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b2043-16e4-49f2-bef0-cd7bd4f846da None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:38.913370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b1221c-1485-44e1-8829-012047ffda18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:38.913983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b1221c-1485-44e1-8829-012047ffda18 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.914262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b1221c-1485-44e1-8829-012047ffda18 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.914530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b1221c-1485-44e1-8829-012047ffda18 None admin] PATH_INFO: `/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.915622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b1221c-1485-44e1-8829-012047ffda18 None admin] RBAC: Authorizing `identity:get_project(project_id=7ca9ceea147b4c7ead1cab8c9dcefc38)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:38.917058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b1221c-1485-44e1-8829-012047ffda18 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:38.918642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 590/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/projects/7ca9ceea147b4c7ead1cab8c9dcefc38 => generated 361 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:38.924714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.971205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:39.019837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480adb73-2558-4161-be26-e31ab31e20a9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.051769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.052355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.052629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.052920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.053625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=7ca9ceea147b4c7ead1cab8c9dcefc38, user.id=53d0274239d44436946eef68ccbc986c) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:39.053927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.055359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98369f1d-7735-442a-8022-54e4e9dbd7b0 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.060795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 610/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/role_assignments?user.id=53d0274239d44436946eef68ccbc986c&scope.project.id=7ca9ceea147b4c7ead1cab8c9dcefc38&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:48.909407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=70 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:49.058294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=44 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:34.463406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e7c90-db47-47e3-a931-c2624f6108f1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:34.464093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e7c90-db47-47e3-a931-c2624f6108f1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:34.464093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e7c90-db47-47e3-a931-c2624f6108f1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:34.465726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 591/1201] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:01:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:34.476225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:34.476860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:34.476860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:34.502564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:34.504094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:34.598372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 611/1202] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:34.903344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225e7c90-db47-47e3-a931-c2624f6108f1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:35.026989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225e7c90-db47-47e3-a931-c2624f6108f1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:35.063910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:35.064817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:35.065294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:35.065723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:35.078320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:35.079826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:35.103633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:35.104779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:35.106427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:35.106427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:35.106427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:35.106427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 592/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:35.111160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:35.249538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d596e7-4d9c-4b7f-a8a1-12c0468c6215 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:35.290192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:35.297949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:35.298633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:35.299103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:35.299618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:35.300265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:35.301313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:35.302743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5905a2d6-570a-4969-9389-c2bbbabbf50f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:35.316618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 612/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:35.322733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:35.385529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:35.438154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add1eb8-cf72-453e-a0cd-b45770e87f35 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:35.476660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f06994-9e04-457d-b94e-cee676b6f83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:35.476660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f06994-9e04-457d-b94e-cee676b6f83d None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:35.477462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f06994-9e04-457d-b94e-cee676b6f83d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:35.477462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f06994-9e04-457d-b94e-cee676b6f83d None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:35.477780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f06994-9e04-457d-b94e-cee676b6f83d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:35.480776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f06994-9e04-457d-b94e-cee676b6f83d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:35.509103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 593/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Sep 29 02:01:35 2022] POST /identity/v3/users => generated 251 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:01:38.028056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b5e75-79c1-4c33-a517-f2bfe9fd207e None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:38.028651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b5e75-79c1-4c33-a517-f2bfe9fd207e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:38.028651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b5e75-79c1-4c33-a517-f2bfe9fd207e None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:38.029942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 613/1206] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:01:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:38.036545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:38.036994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:38.037784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:38.060000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:38.061473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:38.140543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 594/1207] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:38.406153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b5e75-79c1-4c33-a517-f2bfe9fd207e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:38.514382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b5e75-79c1-4c33-a517-f2bfe9fd207e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:38.548828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:38.549475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:38.549749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:38.550013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:38.562126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:38.567959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:38.569284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:38.570155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 614/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 02:01:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:38.574467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:38.624204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:38.692123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7166be-00d4-4f5c-ae0b-fbf63b12c50b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:38.733559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:38.734352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:38.734710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:38.735023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:38.735744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:38.736163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:38.737659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:38.751328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 595/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 02:01:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:38.752934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:38.809800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:38.876104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6e6ed-4832-4611-b2c8-6016b35abf88 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:38.926499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:38.927311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:38.927712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:38.928096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:38.940416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:38.942061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:38.960301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:38.961677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:38.967774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:38.967774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:38.967774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:38.967774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 615/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:38.968076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:39.086620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:39.092604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d694f0-f889-4739-aeb2-3339d3a99d02 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:39.135937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:39.137026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:39.137316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:39.137599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:39.138542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:39.138892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:39.141187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:39.157267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 596/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:39.164948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:39.286408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86832eb-b79b-46ca-9b05-5fc39a34f2e1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:39.291261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:39.335696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:39.336341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:39.336780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:39.337399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:39.347108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:39.348707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:01:39.361991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:39.363076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:39.364399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 616/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:01:39.371146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:39.492861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b116932-d3ee-4b04-820e-ac568e13a4d5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:39.544669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:39.545613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:39.545990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:39.546611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:39.548024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:39.548509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:39.550001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:39.570015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 597/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 29 02:01:39 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:39.575859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:39.700741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93f15b8-1145-4d0f-978d-023a3c91ebd8 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:39.739336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:39.740374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:39.740946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:39.741438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:39.783450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:39.836542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:39.838330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:39.965154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 617/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 29 02:01:39.971071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:39.994914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.087660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc21519-4422-4a86-8609-1708925434f4 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.126190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.126886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.127119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.127384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.138476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.139929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:40.155344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:40.156577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:40.157592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:40.157592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:40.157592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:40.157592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 598/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:01:40.163224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.245914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.290258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05458c5c-1744-4980-b9dd-46bffca9b1ac None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.327624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.328399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.328642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.328965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.329625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:40.329918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.331547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.345026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 618/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:40.351591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.425585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.457409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed0b7e6-4b73-41f1-9b55-2a7418a87ea8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.492610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.493296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.493577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.493849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.500763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.502226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:01:40.513273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:40.514354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:40.515911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 599/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:40.520533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.642779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dba450-daf1-4d20-86ab-d148b0eccab5 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.679485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.694790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.695726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.696138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.696452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.697527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:40.698000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.700043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1969343c-8325-49c7-a59e-0ff18e904d00 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.716912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 619/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:40.722655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.838678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb57d45-d2d9-4b25-9631-fdd2a9167f08 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.871754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.884892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.884892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.884892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.884892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.885345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=f929f94af045445bb0a608dee02a7c5e) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:40.885345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.885345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04663492-8092-4db1-819e-7c8fbfa033ba None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.903136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 600/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/role_assignments?user.id=f929f94af045445bb0a608dee02a7c5e&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:43.092536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335d68c4-5a96-4ab7-827c-58562155e832 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.093106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335d68c4-5a96-4ab7-827c-58562155e832 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.093106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335d68c4-5a96-4ab7-827c-58562155e832 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.094474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 620/1220] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:01:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:43.100188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.100626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.100806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.120757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:43.121981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:43.194018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 601/1221] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:43.459970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335d68c4-5a96-4ab7-827c-58562155e832 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:43.566768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335d68c4-5a96-4ab7-827c-58562155e832 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:43.613438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:43.614243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.614660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.614943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.628318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:43.635687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:43.636614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:43.637648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 621/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:43.643755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:43.754399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98df94f0-10fa-4d6a-ad81-ed7632ca0f04 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:43.780960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:43.793224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:43.794001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.794298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.794747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.795463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:43.795808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:43.797220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:43.806359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 602/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:43.812737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:43.862451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:43.920447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba57189-4ef9-4f84-b5c7-c3ac8fbea005 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:43.960628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:43.961301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.961579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.961847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.971928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:43.973415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:43.988477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:43.989734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:43.990850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:43.990850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:43.990850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:43.990850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 622/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:43.995384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.095786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de968c2c-ba35-4341-b717-fb9242b6e7f8 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.131220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.131901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.132199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.132499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.133206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:44.133510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.134964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.147025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 603/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:44.152914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.267476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:44.275429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec4e77-85d5-4106-a13c-6d0bec066edf None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.312463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.313157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.313442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.313776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.322803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.324307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:01:44.337118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:44.338364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:44.339894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 623/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:44.344279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.393904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:44.450300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dcca3-5a83-4ae6-9b6f-aeffbc4a8b99 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.488358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.489137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.489473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.489770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.491199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:44.491596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.493176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.515902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 604/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 29 02:01:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:44.517280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.633179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c089c42-d99c-42c7-8380-cea4893d4063 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.668471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.669058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.669333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.669602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.680638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.682161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:44.696517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:44.697555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:44.698478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:44.698478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:44.698478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:44.698478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 624/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:44.703535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.802252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361442b7-a3a0-4dd2-8963-c98c96887da3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.804595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:44.839562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.840163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.840442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.840718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.841393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:44.841699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.843054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.853947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 605/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:44.860243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.870297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:44.966669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2f0ae-ddf1-41de-82b2-b1a21d34f31f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:45.009521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:45.010354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:45.010810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:45.011161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:45.018436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:45.020041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:01:45.033818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:45.034985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:45.035895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 625/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 02:01:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:45.042176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:45.157429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc08a5ec-6946-4365-aaa7-4fb925a8f8a7 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:45.195218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:45.195821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:45.196099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:45.196366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:45.197096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:45.197397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:45.198812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2a91ca-6559-4455-9e4c-587ec7294d68 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:45.216656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 606/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 02:01:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:45.218073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:45.293511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:45.329165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b0ce6-846d-4c84-865b-c7f5d52a3218 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:45.366117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde024ee-5b03-43bf-beb5-d06962e5b7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:45.366946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde024ee-5b03-43bf-beb5-d06962e5b7c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:45.367222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde024ee-5b03-43bf-beb5-d06962e5b7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:45.367495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde024ee-5b03-43bf-beb5-d06962e5b7c2 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/f929f94af045445bb0a608dee02a7c5e/roles/5aeb96bfd6ce47eb873387659aedae36` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:45.383579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde024ee-5b03-43bf-beb5-d06962e5b7c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=f929f94af045445bb0a608dee02a7c5e, role_id=5aeb96bfd6ce47eb873387659aedae36)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:45.385069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde024ee-5b03-43bf-beb5-d06962e5b7c2 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:45.399168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 626/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 02:01:45 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/f929f94af045445bb0a608dee02a7c5e/roles/5aeb96bfd6ce47eb873387659aedae36 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:01:47.786204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c230bd1b-325d-4ca6-8507-835f739ab0df None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.786758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c230bd1b-325d-4ca6-8507-835f739ab0df None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.786758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c230bd1b-325d-4ca6-8507-835f739ab0df None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.788637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 607/1233] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:01:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:47.794291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.794730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.794846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.815369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:47.816710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:47.891858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 627/1234] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:48.189292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c230bd1b-325d-4ca6-8507-835f739ab0df None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:48.218027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:48.309118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c230bd1b-325d-4ca6-8507-835f739ab0df None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:48.348532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:48.349277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:48.349575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:48.349852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:48.360893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:48.366650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:48.367673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:48.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 608/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Sep 29 02:01:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:48.374312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:48.427435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:48.485144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabafeca-28e0-4393-bdb1-0ffd464bb605 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:48.518845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:48.519454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:48.519730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:48.520004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:48.520666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] RBAC: Adding query filter params (name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:48.520998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:48.522880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:48.531169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 628/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Sep 29 02:01:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:48.537583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:48.627542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:48.645887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74623b96-a23a-481e-95c1-1f39d0b20578 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:48.683905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:48.684553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:48.684901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:48.685212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:48.696681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:48.698336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:48.716032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:48.717520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:48.718626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:48.718626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:48.718626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:48.718626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 609/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:48.723645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:48.831990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50187e3f-3d7c-4428-8e78-172a895689ad None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:48.842940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:48.872171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:48.872782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:48.873086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:48.873358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:48.874162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:48.874476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:48.876087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:48.889948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 629/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:48.897344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:49.013692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e7bf8e-f676-4d79-8bd9-691c735ea8f2 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:49.048576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:49.049323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:49.049624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:49.049894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:49.053273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:49.058870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:49.060340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:01:49.072246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:49.073447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:49.074428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 610/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:49.079438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:49.187643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7923c93-5a33-487a-8303-8735617df095 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:49.225285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:49.226133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:49.226413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:49.226697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:49.227494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:49.227793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:49.229237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:49.248852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 630/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 29 02:01:49 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:49.254865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:49.369186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fea7cc-1d16-4ae6-8d82-16fba8445f4f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:49.406064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:49.406701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:49.406987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:49.407261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:49.478920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:49.485982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:49.487420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:49.603135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 611/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:49.604592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:49.636017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:49.715542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaba40c0-4493-4426-adad-64c0b4ae5cc1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:49.752058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:49.752707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:49.753015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:49.753287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:49.763736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:49.765327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:49.782190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:49.783513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:49.784402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:49.784402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:49.784402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:49.784402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 631/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:49.789802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:49.889972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:49.897574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4f91bd-56a6-4e40-b9a9-a7d8f0c60ad3 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:49.934312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:49.935022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:49.935295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:49.935620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:49.936374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:49.936708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:49.938271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:49.950496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 612/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:49.956092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:50.039784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:50.063841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f82df1-96d9-4e24-b322-eab398e35de9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:50.100653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:50.101323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:50.101600 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:50.101864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:50.109796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:50.111236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:01:50.122572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:50.123751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:50.124785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 632/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 02:01:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:50.129716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:50.244698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e745efb-a699-4f8a-8db7-b4fe541489ea None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:50.286220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:50.286852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:50.287126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:50.287396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:50.288079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:50.288379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:50.290009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd96c4d9-3ba6-4d16-ab0e-cbeb14278676 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:50.304024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 613/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 02:01:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:50.311569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:50.425529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5289e994-8f71-417e-9511-fa348fc7ae9d None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:50.464477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:50.465156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:50.465436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:50.465697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:50.466541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] RBAC: Adding query filter params (role.id=5aeb96bfd6ce47eb873387659aedae36, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=f929f94af045445bb0a608dee02a7c5e) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:50.466851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:50.468302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab3400f-48a2-4782-8271-1b5b2e4f7928 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:50.474863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 633/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:50 2022] GET /identity/v3/role_assignments?user.id=f929f94af045445bb0a608dee02a7c5e&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=5aeb96bfd6ce47eb873387659aedae36 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:52.716276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62935d6-9b4d-4578-bf5d-30d64eea4d33 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:52.716901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62935d6-9b4d-4578-bf5d-30d64eea4d33 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:52.717014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62935d6-9b4d-4578-bf5d-30d64eea4d33 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:52.718390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 614/1247] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:01:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:01:52.723746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:52.724165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:52.724340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:52.731328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:52.743793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:52.745019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:52.813404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 634/1248] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:53.089398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62935d6-9b4d-4578-bf5d-30d64eea4d33 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:53.108560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:53.194903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62935d6-9b4d-4578-bf5d-30d64eea4d33 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:53.236818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:53.237464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:53.237797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:53.238115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:53.252181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:53.257945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:53.260949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:53.262056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 615/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:53.265282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:53.368321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d054d413-5bd8-4dc7-9747-ce10df4ec936 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:53.398995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:53.404356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:53.404992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:53.405272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:53.405540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:53.406200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:53.406491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:53.407874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:53.416165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 635/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:53.422125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:53.501483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:53.525512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0189c-2f62-420a-a4d6-c16e5728c334 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:53.559911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:53.560535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:53.560819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:53.561117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:53.573317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:53.574766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:53.589777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:53.590878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:53.591907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:53.591907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:53.591907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:53.591907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 616/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:53.596718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:53.694315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf79c71a-7fa7-4204-a94a-6eb53ea9e1ca None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:53.726637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:53.727229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:53.727506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:53.727773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:53.728400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:53.728697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:53.730102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:53.741306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 636/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:53.746950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:53.841769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075bea70-7184-4c02-9b4d-ea4d39166fe6 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:53.865835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:53.873737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:53.874350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:53.874626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:53.874896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:53.882476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:53.883879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:01:53.894475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:53.895402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:53.897317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 617/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:53.901739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:53.955087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:53.997654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466b4bd6-8e1e-46b8-b3d9-4afdaa508d2a None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:54.030205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:54.030794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:54.031067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:54.031332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:54.032039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:54.032336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:54.033771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:54.049800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 637/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 29 02:01:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:54.055486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:54.151889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82d3479-2b95-4b8e-ac5f-78212906c7a5 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:54.184138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:54.184724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:54.185048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:54.185316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:54.242357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:54.254533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:54.256586 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:54.348214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 618/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:54 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:54.354015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:54.449570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ccc60e-46cc-4033-8a17-5559a243cf23 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:54.462451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:54.483302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:54.483923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:54.484203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:54.484478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:54.494163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:54.495543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:54.509615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:54.510727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:54.511825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:54.511825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:54.511825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:54.511825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 638/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:54.516426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:54.602262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:54.614106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fad80-185b-4e8f-86cb-ec67d9dcbb9d None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:54.647600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:54.648192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:54.648473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:54.648744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:54.649410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:54.649711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:54.651090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:54.661810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 619/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:54.667589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:54.762609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc43475-bed2-4206-8760-d014d020a4b1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:54.794005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:54.794608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:54.794881 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:54.795145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:54.801434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:54.802887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:01:54.814389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:54.815450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:01:54.816379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 639/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 02:01:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:54.822016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:54.915395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545fa8f5-b2dc-487a-931c-cc2fd650d53c None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:54.949731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:54.950533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:54.950811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:54.951078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:54.951759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:54.952058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:54.953526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551f4ee5-db55-4341-9210-e7b451167b57 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:54.966071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 620/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 02:01:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:54.972230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:54.997209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:55.069637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0496673b-424c-445d-8c7a-077f8cb69bb3 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:55.100969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:55.101550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:55.101824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:55.102089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:55.102789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=f929f94af045445bb0a608dee02a7c5e) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:55.103085 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:55.104496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6f3616-801f-48a7-ac64-c45e2371c083 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:55.109645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 640/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:54 2022] GET /identity/v3/role_assignments?user.id=f929f94af045445bb0a608dee02a7c5e&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=619e79e7bb94488780bd4da040d61c34 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:57.230469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c392ab-f04a-4066-9d9e-75d2211688e6 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:57.231021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c392ab-f04a-4066-9d9e-75d2211688e6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:57.231298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c392ab-f04a-4066-9d9e-75d2211688e6 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:57.232941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 621/1261] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:01:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:57.239058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:57.239455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:57.239606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:57.257466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:57.258634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:57.323033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 641/1262] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:01:57.587678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c392ab-f04a-4066-9d9e-75d2211688e6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:57.598395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:57.689818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c392ab-f04a-4066-9d9e-75d2211688e6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:57.722728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:57.723341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:57.723620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:57.723889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:57.733459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:57.738120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:57.739187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:57.740025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 622/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:57.745223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:57.763007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:57.841318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0c7ef3-fd63-4732-967d-4d585b5ab999 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:57.873320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:57.873918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:57.874199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:57.874468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:57.875118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:57.875420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:57.876769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:57.885026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 642/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:57.890601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:57.954825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:57.986677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fdf49b-418a-4d50-82c4-8168d36fd6c4 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.019335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.019945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.020221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.020486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.029915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.031339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:58.045065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:58.046156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:58.047054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:58.047054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:58.047054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:58.047054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 623/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:58.052018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:58.129759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:58.148608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf54a76-a537-4b74-bcdb-d7cb6a88d979 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.180661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.181280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.181562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.181836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.182467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:58.182771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.184127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.196127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 643/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:58.202487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:58.301615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26b099f-de64-422d-a32c-3bfc5466e283 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.320808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:58.334329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.334945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.335221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.335490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.343292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.344723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:01:58.355884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:58.357023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:58.358247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 624/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:58.362879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:58.458783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992235a-c431-48c6-b96e-9f0641d4c540 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.491274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.491858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.492133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.492400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.493139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:58.493439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.494848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.502975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:58.512429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 644/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:58.518126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:58.613016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fd4202-5751-4fad-9640-cfdd5e946c6e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.644648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.645286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.645567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.645837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.656227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.657680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:58.671560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:58.672657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:01:58.673609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:58.673609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:01:58.673609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:58.673609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 625/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:58.678325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:58.774066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8b26f1-f011-4118-88e8-ff4892072c9f None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.806456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.807045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.807321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.807596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.808221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:58.808520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.809933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.820923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 645/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:58.826594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:58.857317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:58.923193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1a2e8-537a-4475-aeb4-170735299b5e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:58.956113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:58.956723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:58.957036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:58.957311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:58.963667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:58.965117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:01:58.975951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:58.976982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:01:58.978097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 626/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:58.982825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:59.038727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:59.080462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fdc5bc-89e5-4b39-a03d-2475c0622352 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:59.112098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:59.112684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:59.112999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:59.113272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:59.113946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:59.114247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:59.115687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ba5c18-ed9a-49ce-97ed-2e82dd3fb94f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:59.126960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 646/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 02:01:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:59.133417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:59.227869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:59.236766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643051e5-9d19-4944-a800-a57bfd7fe698 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:59.268438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fee855d-f7ef-4d6a-bbf5-ef1731e2b7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:59.269096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fee855d-f7ef-4d6a-bbf5-ef1731e2b7f4 None admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:59.269374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fee855d-f7ef-4d6a-bbf5-ef1731e2b7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:59.269641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fee855d-f7ef-4d6a-bbf5-ef1731e2b7f4 None admin] PATH_INFO: `/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/f929f94af045445bb0a608dee02a7c5e/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:59.271481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fee855d-f7ef-4d6a-bbf5-ef1731e2b7f4 None admin] RBAC: Authorizing `identity:create_grant(project_id=2fb848d359fb4361b7724b736e0f5333, user_id=f929f94af045445bb0a608dee02a7c5e, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:59.272915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fee855d-f7ef-4d6a-bbf5-ef1731e2b7f4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:59.285525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 627/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Sep 29 02:01:59 2022] PUT /identity/v3/projects/2fb848d359fb4361b7724b736e0f5333/users/f929f94af045445bb0a608dee02a7c5e/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:02:01.377875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4944b729-e541-4526-b192-91038f35068e None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:01.378437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4944b729-e541-4526-b192-91038f35068e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:01.378437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4944b729-e541-4526-b192-91038f35068e None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:01.379724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 647/1274] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:02:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:01.384840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:01.385226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:01.385431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:01.403251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:01.404442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:01.468548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 628/1275] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:01.735844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4944b729-e541-4526-b192-91038f35068e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:01.831713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4944b729-e541-4526-b192-91038f35068e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:01.849142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:01.866051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:01.866671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:01.866946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:01.867214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:01.876714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:02:01.881461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:01.882582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:01.883883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 648/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Sep 29 02:02:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:01.888444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:01.965742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:01.984096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d4e75f-b13d-4ab1-9e78-6a6dc1f394c4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:02.016356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:02.016973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:02.017256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:02.017529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:02.018188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:02.018492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:02.019844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:02.027975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 629/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Sep 29 02:02:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:02.033640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:02.129856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139d1e9a-7602-4de3-b693-63998346a740 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:02.162751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:02.163365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:02.163639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:02.163908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:02.173409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:02.174833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:02.188560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:02.189554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:02.190522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:02.190522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:02.190522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:02.190522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 649/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:02.196118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:02.309693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e799f4-e7b3-480f-9fca-d2cf894602b4 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:02.342270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:02.342869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:02.343148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:02.343418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:02.344046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:02.344342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:02.345716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:02.356831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 630/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:02.362534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:02.393062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:02.459698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f69c17e-c307-49ac-a12b-cbc7b08ed762 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:02.491274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:02.491880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:02.492155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:02.492424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:02.499394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:02.500811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:02:02.512063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:02.513199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:02.514468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 650/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:02.518851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:02.522613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:02.618141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9477c-7676-4bd5-9ac9-1bf522b9d3fe None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:02.650903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:02.651485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:02.651762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:02.652029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:02.652735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:02.653059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:02.654460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:02.671151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 631/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Sep 29 02:02:02 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:02.676978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:02.763668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:02.773114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3d952c-03a5-48b7-91e0-e74661c4c156 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:02.805301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:02.805885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:02.806164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:02.806434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:02.869907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:02.871196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:02.956396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:02.966521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 651/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Sep 29 02:02:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:02:02.972281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:03.029930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:03.067470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82ed9ae-2499-45bc-9008-c57199c3796f None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:03.101401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:03.102022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:03.102306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:03.102579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:03.112139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:03.113602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:03.127614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:03.128722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:03.129883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:03.129883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:03.129883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:02:03.129883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 632/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:03.134824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:03.235409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b63640-c625-45ea-a80d-a39c2b79fdd9 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:03.267723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:03.268320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:03.268597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:03.268890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:03.269524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:03.269823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:03.271687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:03.282418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 652/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:03.288834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:03.385967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5016ba6a-f6d1-4772-8e8f-9b34453e3a8e None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:03.399190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:03.419920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:03.420545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:03.420823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:03.421122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:03.427499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:03.428944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:02:03.440569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:03.441619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: ERROR keystone.server.flask.application Sep 29 02:02:03.442735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 633/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Sep 29 02:02:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:03.447746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:03.495908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:03.544477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5111e17-f1a2-476c-a4e5-cef1f511bd18 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:03.576980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:03.577578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:03.577856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:03.578125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:03.578804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:03.579105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:03.580518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500efc0-8fe8-4cd8-9388-8f0882590b69 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:03.593339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 653/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Sep 29 02:02:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:03.599321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:03.694236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b82e71e-29c9-4af7-9c52-cc6a1ad82869 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:03.725701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:03.726281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:03.726559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:03.726828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:03.727530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] RBAC: Adding query filter params (role.id=619e79e7bb94488780bd4da040d61c34, scope.project.id=2fb848d359fb4361b7724b736e0f5333, user.id=f929f94af045445bb0a608dee02a7c5e) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:03.727833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:03.729267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94502823-fdf1-4347-a62f-fb45f6a5bed6 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:03.734709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 634/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Sep 29 02:02:03 2022] GET /identity/v3/role_assignments?user.id=f929f94af045445bb0a608dee02a7c5e&scope.project.id=2fb848d359fb4361b7724b736e0f5333&role.id=619e79e7bb94488780bd4da040d61c34 => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:05.859656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679df4c5-b0b0-48ee-8a78-0d35ee116f4c None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:05.860154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679df4c5-b0b0-48ee-8a78-0d35ee116f4c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:05.860154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679df4c5-b0b0-48ee-8a78-0d35ee116f4c None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:05.861479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 654/1288] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:02:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:05.866646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:05.867061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:05.867204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:05.885461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:05.886651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:05.915094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:05.951333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 635/1289] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:06.220257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679df4c5-b0b0-48ee-8a78-0d35ee116f4c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:06.326654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679df4c5-b0b0-48ee-8a78-0d35ee116f4c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:06.336793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:06.358880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:06.359497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:06.359773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:06.360041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:06.360611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:06.361990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:06.370961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:06.372241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:06.373378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 655/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 02:02:06 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:02:06.378035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:06.471946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55620fc1-5a41-48de-83ed-458f857988a3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:06.503864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:06.504455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:06.504733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:06.505032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:06.505709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:06.506010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:06.507357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a15daa-fb24-4f67-8614-65335f2b0a0f None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:06.511896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 636/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 29 02:02:06 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:06.517370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:06.613574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171cd472-9d22-4c03-b71f-d941c9234a56 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:06.645565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:06.646746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:06.647020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:06.647291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] PATH_INFO: `/v3/services` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:06.647902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] RBAC: Adding query filter params (type=placement) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:06.648198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:06.649586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803ae225-2b39-42a0-8fc2-4a9560deacfd None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:06.653866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 656/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 29 02:02:06 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:08.750950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a73f24-ded3-4888-b815-b48f455efbf1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:08.751507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a73f24-ded3-4888-b815-b48f455efbf1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:08.751507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a73f24-ded3-4888-b815-b48f455efbf1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:08.752795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 637/1293] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:02:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:08.758395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86654ee4-9020-4330-9733-9ece166ff839 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:08.758763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86654ee4-9020-4330-9733-9ece166ff839 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:08.758942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86654ee4-9020-4330-9733-9ece166ff839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:08.776507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-86654ee4-9020-4330-9733-9ece166ff839 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:08.777771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-86654ee4-9020-4330-9733-9ece166ff839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:08.806171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:08.841757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 657/1294] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:09.104809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a73f24-ded3-4888-b815-b48f455efbf1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:09.136309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:09.206379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a73f24-ded3-4888-b815-b48f455efbf1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:09.239131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5f5939-ae54-43bb-ab0a-056850f3ca72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:09.239757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5f5939-ae54-43bb-ab0a-056850f3ca72 None admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:09.240035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5f5939-ae54-43bb-ab0a-056850f3ca72 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:09.240304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5f5939-ae54-43bb-ab0a-056850f3ca72 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:09.240976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5f5939-ae54-43bb-ab0a-056850f3ca72 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:09.242317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5f5939-ae54-43bb-ab0a-056850f3ca72 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:09.255828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 638/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Sep 29 02:02:09 2022] POST /identity/v3/services => generated 246 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:02:11.392588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaf9039-2a5e-4ab1-836d-762920741cd5 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:11.393189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaf9039-2a5e-4ab1-836d-762920741cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:11.393189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeaf9039-2a5e-4ab1-836d-762920741cd5 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:11.394500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 658/1296] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:02:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:11.401125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:11.401505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:11.401674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:11.419447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:11.420625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:11.490354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 639/1297] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:11 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:11.761543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaf9039-2a5e-4ab1-836d-762920741cd5 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:11.859840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeaf9039-2a5e-4ab1-836d-762920741cd5 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:11.895367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:11.896024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:11.896304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:11.896576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] PATH_INFO: `/v3/services/placement` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:11.897183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:11.898563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:11.907223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:11.908380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:11.909260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 659/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 02:02:11 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:11.914989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:11.932984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:12.010059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17561300-8e4c-4ca9-8753-bac063d005b2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:12.042867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:12.043407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:12.043687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:12.043958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:12.044611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:12.044938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:12.046291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:12.051175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 640/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 29 02:02:11 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:12.057623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:12.087822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:12.152987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5accbf-4be8-48c5-acd8-cf853b2685ff None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:12.184553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:12.185162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:12.185442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:12.185712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:12.186669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] RBAC: Adding query filter params (interface=public, service_id=502754893f944e2ebf85e3ee4c196107, region_id=RegionOne) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:12.186974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:12.188331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e532991a-b59e-4384-b675-7cc9d4d080df None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:12.194852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 660/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Sep 29 02:02:12 2022] GET /identity/v3/endpoints?service_id=502754893f944e2ebf85e3ee4c196107&interface=public®ion_id=RegionOne => generated 194 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:12.200760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:12.293617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:12.311295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fd20af-5b69-4db1-b5ed-cec4f68f0859 None admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:12.343738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc2dab5-5c69-4dce-8090-c4a87e79a349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:12.344332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc2dab5-5c69-4dce-8090-c4a87e79a349 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:12.344608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc2dab5-5c69-4dce-8090-c4a87e79a349 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:12.344898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc2dab5-5c69-4dce-8090-c4a87e79a349 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:12.345497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc2dab5-5c69-4dce-8090-c4a87e79a349 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:12.346846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc2dab5-5c69-4dce-8090-c4a87e79a349 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:12.353942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 641/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Sep 29 02:02:12 2022] GET /identity/v3/services => generated 2191 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:02:14.447008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6d814-4d07-4471-8d40-25fc84acee68 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:14.447534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6d814-4d07-4471-8d40-25fc84acee68 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:14.447534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6d814-4d07-4471-8d40-25fc84acee68 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:14.448900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 661/1302] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:02:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:14.454034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:14.454430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:14.454582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:14.472473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:14.473676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:14.535364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 642/1303] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:14 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:14.798124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6d814-4d07-4471-8d40-25fc84acee68 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:14.890671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:14.897464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6d814-4d07-4471-8d40-25fc84acee68 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:14.930024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:14.930641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:14.930918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:14.931187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:14.931758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:14.933139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:14.941612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:14.942790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: ERROR keystone.server.flask.application Sep 29 02:02:14.943661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 662/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Sep 29 02:02:14 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:14.948853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:15.020530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:15.042677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a454fe6-9f11-492a-bf6d-eafb32e97f71 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:15.075110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:15.075650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:15.075930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:15.076200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] PATH_INFO: `/v3/services` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:15.076815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:15.077143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:15.078479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dbe74c-94da-41a8-9d25-598f3ddf3e90 None admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:15.083413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 643/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Sep 29 02:02:14 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:15.088972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:15.184767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a375a0-5be7-492b-872e-4516442f76a1 None admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:15.217201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa85a505-a43f-44fe-96e2-9d2a5d90473f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:15.217790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa85a505-a43f-44fe-96e2-9d2a5d90473f None admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:15.218065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa85a505-a43f-44fe-96e2-9d2a5d90473f None admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:15.218333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa85a505-a43f-44fe-96e2-9d2a5d90473f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:15.218985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa85a505-a43f-44fe-96e2-9d2a5d90473f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:15.220335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa85a505-a43f-44fe-96e2-9d2a5d90473f None admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:15.239726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 663/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:02:15 2022] POST /identity/v3/endpoints => generated 331 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:02:25.080438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=30 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:25.237468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:18.620571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1094a-4f18-4dae-a8d9-8d1ceb94bde9 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:18.621076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1094a-4f18-4dae-a8d9-8d1ceb94bde9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:18.621283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd1094a-4f18-4dae-a8d9-8d1ceb94bde9 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:18.627631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a072a731-9a2e-4c5a-91c8-aa3d6a4137bc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:18.628015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a072a731-9a2e-4c5a-91c8-aa3d6a4137bc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:18.628172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a072a731-9a2e-4c5a-91c8-aa3d6a4137bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:18.633067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 644/1307] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:18 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:18.716982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a072a731-9a2e-4c5a-91c8-aa3d6a4137bc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:18.718457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a072a731-9a2e-4c5a-91c8-aa3d6a4137bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:18.885729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 664/1308] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:03:19.171000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161e920-b3ca-47d0-9286-387bd1088487 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:19.171669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161e920-b3ca-47d0-9286-387bd1088487 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:19.172154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8161e920-b3ca-47d0-9286-387bd1088487 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:19.174030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 645/1309] 10.222.0.27 () {54 vars in 964 bytes} [Thu Sep 29 02:03:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:19.183157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da64a5fd-809d-4441-9ead-eecc42a6eebe None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:19.183438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da64a5fd-809d-4441-9ead-eecc42a6eebe None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:19.183712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da64a5fd-809d-4441-9ead-eecc42a6eebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:19.290919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-da64a5fd-809d-4441-9ead-eecc42a6eebe None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:19.292361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-da64a5fd-809d-4441-9ead-eecc42a6eebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:19.317388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:19.384197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 665/1310] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Sep 29 02:03:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:19.394554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161e920-b3ca-47d0-9286-387bd1088487 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:19.396384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8161e920-b3ca-47d0-9286-387bd1088487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:19.541631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8161e920-b3ca-47d0-9286-387bd1088487 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:19.636293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:19.637910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:19.638487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:19.639009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:19.640796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:19.706221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:19.773073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:19.774990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bb64ea-62d4-482f-bb11-6b893fa3486c service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:19.797655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 646/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:22.307543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d796a8-18df-4dec-b585-cf663953539a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:22.308119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d796a8-18df-4dec-b585-cf663953539a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:22.308119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d796a8-18df-4dec-b585-cf663953539a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:22.309468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 666/1312] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:22.314753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f12801-7b37-4dae-84eb-67bd70a3c1e4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:22.315181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f12801-7b37-4dae-84eb-67bd70a3c1e4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:22.315292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f12801-7b37-4dae-84eb-67bd70a3c1e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:22.394425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-11f12801-7b37-4dae-84eb-67bd70a3c1e4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:22.395607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-11f12801-7b37-4dae-84eb-67bd70a3c1e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:22.488535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 647/1313] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:22.770522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d796a8-18df-4dec-b585-cf663953539a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:22.772051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-51d796a8-18df-4dec-b585-cf663953539a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:22.872552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d796a8-18df-4dec-b585-cf663953539a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:22.941721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:22.944459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:22.945232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:22.945700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:22.946126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:22.947907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:23.052342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:23.053721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a216d2f5-f2b7-4c2d-b09c-5cbedc5184ca service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:23.071026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 667/1314] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:25.491440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16436e97-30f0-4d5f-b7a2-69c193dad2bd None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:25.492001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16436e97-30f0-4d5f-b7a2-69c193dad2bd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:25.492001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16436e97-30f0-4d5f-b7a2-69c193dad2bd None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:25.493335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 648/1315] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:25.498578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65f7b8-262b-4800-8edc-d0a92406265f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:25.498979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65f7b8-262b-4800-8edc-d0a92406265f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:25.499125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65f7b8-262b-4800-8edc-d0a92406265f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:25.563713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8b65f7b8-262b-4800-8edc-d0a92406265f None None] MFA Rules not processed for user `b2a267a3bc0a40988be575ff351b6ef4`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:25.564946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8b65f7b8-262b-4800-8edc-d0a92406265f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:25.616470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:25.660304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 668/1316] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:25 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:25.949873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16436e97-30f0-4d5f-b7a2-69c193dad2bd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:25.951388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-16436e97-30f0-4d5f-b7a2-69c193dad2bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:25.960977 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:26.052767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16436e97-30f0-4d5f-b7a2-69c193dad2bd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:26.125209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:26.125800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:26.126079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:26.126347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:26.127915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:26.187862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:26.233576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:26.234950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998dea6-bc37-44d3-b1c1-8055a4321273 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:26.253283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 649/1317] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:25 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:28.951747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37e35c-1389-4ea2-9aa5-cbd8ce230b9e None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:28.952255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37e35c-1389-4ea2-9aa5-cbd8ce230b9e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:28.952255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37e35c-1389-4ea2-9aa5-cbd8ce230b9e None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:28.953574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 669/1318] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:28.959296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1059b8f1-71c7-4f41-b464-36d2c342cf89 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:28.959676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1059b8f1-71c7-4f41-b464-36d2c342cf89 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:28.959859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1059b8f1-71c7-4f41-b464-36d2c342cf89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:29.023453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1059b8f1-71c7-4f41-b464-36d2c342cf89 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:29.024656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1059b8f1-71c7-4f41-b464-36d2c342cf89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:29.121129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 650/1319] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:28 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:29.404331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f37e35c-1389-4ea2-9aa5-cbd8ce230b9e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:29.405896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3f37e35c-1389-4ea2-9aa5-cbd8ce230b9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:29.509269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f37e35c-1389-4ea2-9aa5-cbd8ce230b9e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:29.579194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:29.579795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:29.580066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:29.580329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:29.582536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:29.591112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:29.690533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:29.692405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49273b20-aec5-4151-a03b-38fdf9378141 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:29.710698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 670/1320] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:32.476382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63133b-cafb-4a73-aafe-8f76424b3964 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:32.476929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63133b-cafb-4a73-aafe-8f76424b3964 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:32.476929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63133b-cafb-4a73-aafe-8f76424b3964 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:32.478201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 651/1321] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:32.483376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a252723-11f9-496a-bb75-6fd1bdfd6b65 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:32.483774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a252723-11f9-496a-bb75-6fd1bdfd6b65 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:32.484053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a252723-11f9-496a-bb75-6fd1bdfd6b65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:32.555073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8a252723-11f9-496a-bb75-6fd1bdfd6b65 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:32.556331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8a252723-11f9-496a-bb75-6fd1bdfd6b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:32.622756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:32.672969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 671/1322] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:32.966898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a8843-368e-464a-86d5-f23d148f7564 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:32.967185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a8843-368e-464a-86d5-f23d148f7564 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:32.967466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062a8843-368e-464a-86d5-f23d148f7564 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:32.968812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 652/1323] 10.222.0.27 () {54 vars in 964 bytes} [Thu Sep 29 02:03:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:32.975596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d8e5c-92b2-4d70-a813-20844d45bb4b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:32.975885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d8e5c-92b2-4d70-a813-20844d45bb4b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:32.976160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d8e5c-92b2-4d70-a813-20844d45bb4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:33.051208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a39d8e5c-92b2-4d70-a813-20844d45bb4b None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:33.052405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a39d8e5c-92b2-4d70-a813-20844d45bb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:33.125699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 672/1324] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Sep 29 02:03:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:33.134899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a8843-368e-464a-86d5-f23d148f7564 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:33.136596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-062a8843-368e-464a-86d5-f23d148f7564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:33.156690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:33.242194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062a8843-368e-464a-86d5-f23d148f7564 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:33.314255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:33.314860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:33.315139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:33.315407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:33.317552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:33.387313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:33.424907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:33.426376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836a81b4-2f84-4750-a6d9-57b36e0e3f48 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:33.445137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 653/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:35.907238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7e63b1-f720-4cb3-abd8-ddc5982fdb2f None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:35.907749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7e63b1-f720-4cb3-abd8-ddc5982fdb2f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:35.907749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7e63b1-f720-4cb3-abd8-ddc5982fdb2f None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:35.909036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 673/1326] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:35.914440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b345f689-181c-4f84-8ca2-5e0fb75e51ad None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:35.914853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b345f689-181c-4f84-8ca2-5e0fb75e51ad None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:35.915013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b345f689-181c-4f84-8ca2-5e0fb75e51ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:35.980267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b345f689-181c-4f84-8ca2-5e0fb75e51ad None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:35.981493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b345f689-181c-4f84-8ca2-5e0fb75e51ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:36.077349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 654/1327] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:03:36.364839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7e63b1-f720-4cb3-abd8-ddc5982fdb2f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:36.366390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ae7e63b1-f720-4cb3-abd8-ddc5982fdb2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:36.425998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:36.467961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7e63b1-f720-4cb3-abd8-ddc5982fdb2f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:36.540273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:36.540945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:36.541173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:36.541446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:36.543290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:36.652649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:36.655042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:36.656498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66c1b1-7341-4f6d-a9df-6f271b79a067 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:36.675739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 674/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:38.908698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbec9759-6bef-42f2-aae3-0572d07602f1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:38.909363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbec9759-6bef-42f2-aae3-0572d07602f1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:38.909481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbec9759-6bef-42f2-aae3-0572d07602f1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:38.910767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 655/1329] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:38.916163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d8e5c-d2aa-420a-b9a1-5884d76575db None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:38.916617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d8e5c-d2aa-420a-b9a1-5884d76575db None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:38.916729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d8e5c-d2aa-420a-b9a1-5884d76575db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:38.985202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-208d8e5c-d2aa-420a-b9a1-5884d76575db None None] MFA Rules not processed for user `b2a267a3bc0a40988be575ff351b6ef4`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:38.986412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-208d8e5c-d2aa-420a-b9a1-5884d76575db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:39.083835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 675/1330] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:38 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:39.368987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbec9759-6bef-42f2-aae3-0572d07602f1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:39.370498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fbec9759-6bef-42f2-aae3-0572d07602f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:39.401320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:39.476708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbec9759-6bef-42f2-aae3-0572d07602f1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:39.552593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:39.553235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:39.553513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:39.553781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:39.555364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:39.646442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:39.683511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:39.685008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a914d6c-ba2f-4d5f-921b-31ca11db67ad service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:39.703964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 656/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:39 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:42.068538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02697c5-6e33-4ee1-ade8-3aa481459523 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:42.069090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02697c5-6e33-4ee1-ade8-3aa481459523 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:42.069186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02697c5-6e33-4ee1-ade8-3aa481459523 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:42.070899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 676/1332] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:42.076249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd84875-93ec-4140-b566-e88ddcdaeca6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:42.076646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd84875-93ec-4140-b566-e88ddcdaeca6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:42.076793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd84875-93ec-4140-b566-e88ddcdaeca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:42.140331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cbd84875-93ec-4140-b566-e88ddcdaeca6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:42.141555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cbd84875-93ec-4140-b566-e88ddcdaeca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:42.236143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 657/1333] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:42.517004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02697c5-6e33-4ee1-ade8-3aa481459523 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:42.519117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a02697c5-6e33-4ee1-ade8-3aa481459523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:42.587235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:42.621381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02697c5-6e33-4ee1-ade8-3aa481459523 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:42.692751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:42.693377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:42.693655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:42.693925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:42.695765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:42.799508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:42.801435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d35c33-959a-4729-b43e-e47798181a15 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:42.811062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:42.819962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 677/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:45.008654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e85956e-8e42-41c7-9f5a-ac2df8d5095d None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:45.009235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e85956e-8e42-41c7-9f5a-ac2df8d5095d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:45.009235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e85956e-8e42-41c7-9f5a-ac2df8d5095d None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:45.010593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 658/1335] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:45.015830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a9a71-b32f-4f07-bb38-a92377920e46 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:45.016221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a9a71-b32f-4f07-bb38-a92377920e46 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:45.016382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87a9a71-b32f-4f07-bb38-a92377920e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:45.080757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d87a9a71-b32f-4f07-bb38-a92377920e46 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:45.082460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d87a9a71-b32f-4f07-bb38-a92377920e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:45.177605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 678/1336] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:45.465841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e85956e-8e42-41c7-9f5a-ac2df8d5095d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:45.467377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0e85956e-8e42-41c7-9f5a-ac2df8d5095d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:45.506530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:45.569510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e85956e-8e42-41c7-9f5a-ac2df8d5095d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:45.640202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:45.640799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:45.641098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:45.641365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:45.642932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:45.723768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:45.748576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:45.750010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90ad64-f3d7-4da6-ad7b-05d361df08d5 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:45.767526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 659/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:48.495310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb07297-37d8-41c0-bc28-dbe55a4caf56 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:48.495820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb07297-37d8-41c0-bc28-dbe55a4caf56 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:48.495936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb07297-37d8-41c0-bc28-dbe55a4caf56 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:48.497215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 679/1338] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:48.502240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d2ad48-f224-40f9-bf8b-64cabbfec9d6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:48.503186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d2ad48-f224-40f9-bf8b-64cabbfec9d6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:48.503288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d2ad48-f224-40f9-bf8b-64cabbfec9d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:48.566981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-06d2ad48-f224-40f9-bf8b-64cabbfec9d6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:48.568218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-06d2ad48-f224-40f9-bf8b-64cabbfec9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:48.662599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 660/1339] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:03:48.939167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb07297-37d8-41c0-bc28-dbe55a4caf56 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:48.941350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7eb07297-37d8-41c0-bc28-dbe55a4caf56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:49.014000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:49.041149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb07297-37d8-41c0-bc28-dbe55a4caf56 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:49.110936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:49.111526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:49.111800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:49.112612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:49.114198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:49.225324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:49.226677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f1ec62-3e66-4985-b04b-97755a3175db service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:49.245488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 680/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:48 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:53.274951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e18f1ba-739d-424c-8ac2-9e9ec24d1d63 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:53.275540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e18f1ba-739d-424c-8ac2-9e9ec24d1d63 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:53.275540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e18f1ba-739d-424c-8ac2-9e9ec24d1d63 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:53.277032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 661/1341] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:53.282219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6356b96b-b350-42ee-bca8-8ca53dc5d8ab None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:53.282665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6356b96b-b350-42ee-bca8-8ca53dc5d8ab None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:53.282747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6356b96b-b350-42ee-bca8-8ca53dc5d8ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:53.288182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:53.358522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6356b96b-b350-42ee-bca8-8ca53dc5d8ab None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:53.360435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6356b96b-b350-42ee-bca8-8ca53dc5d8ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:53.472981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 681/1342] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:53.748410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e18f1ba-739d-424c-8ac2-9e9ec24d1d63 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:53.750073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2e18f1ba-739d-424c-8ac2-9e9ec24d1d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:53.798968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:53.853916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e18f1ba-739d-424c-8ac2-9e9ec24d1d63 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:53.934847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:53.935956 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:53.936241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:53.936513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:53.938137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:54.033169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:54.050263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:54.051692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abecd48-f90d-44eb-9f56-206bf940560d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:54.069169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 662/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:03:56.857446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96332055-4c46-4d43-9f27-976cdcd1d9b7 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:56.857995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96332055-4c46-4d43-9f27-976cdcd1d9b7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:56.857995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96332055-4c46-4d43-9f27-976cdcd1d9b7 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:56.859242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 682/1344] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:03:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:03:56.865413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d07029-aa4a-48cb-b0b4-2c71532dce40 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:56.865831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d07029-aa4a-48cb-b0b4-2c71532dce40 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:56.865970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d07029-aa4a-48cb-b0b4-2c71532dce40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:56.930720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a4d07029-aa4a-48cb-b0b4-2c71532dce40 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:56.931914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a4d07029-aa4a-48cb-b0b4-2c71532dce40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:57.026235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 663/1345] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:56 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:03:57.311857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96332055-4c46-4d43-9f27-976cdcd1d9b7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:57.313514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-96332055-4c46-4d43-9f27-976cdcd1d9b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:57.405440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:57.422514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96332055-4c46-4d43-9f27-976cdcd1d9b7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:57.495051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:57.495649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:57.495926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:57.496197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:57.497815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:57.611095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:57.612576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b392d8-7da7-484b-bd10-959d551bd93a service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:57.630334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 683/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:03:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:01.753990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a799e1f8-6dc2-4836-b034-4e48d6231841 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:01.754505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a799e1f8-6dc2-4836-b034-4e48d6231841 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:01.754617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a799e1f8-6dc2-4836-b034-4e48d6231841 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:01.755863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 664/1347] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:01.761175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69251589-2f72-4c5d-a3d5-935e3b719f36 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:01.761618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69251589-2f72-4c5d-a3d5-935e3b719f36 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:01.761742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69251589-2f72-4c5d-a3d5-935e3b719f36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:01.777745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:01.827808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-69251589-2f72-4c5d-a3d5-935e3b719f36 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:01.829036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-69251589-2f72-4c5d-a3d5-935e3b719f36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:01.922641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 684/1348] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:02.204920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a799e1f8-6dc2-4836-b034-4e48d6231841 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:02.206953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a799e1f8-6dc2-4836-b034-4e48d6231841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:02.266095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:02.312641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a799e1f8-6dc2-4836-b034-4e48d6231841 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:02.386161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:02.386770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:02.387049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:02.387318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:02.388943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:02.499502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:02.504254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:02.505729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac04bb6-3ddf-40a9-8f0b-061e9a0328c5 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:02.525326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 665/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:07.418007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54226066-11bf-4145-ae56-a943f09e716d None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:07.418460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54226066-11bf-4145-ae56-a943f09e716d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:07.418582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54226066-11bf-4145-ae56-a943f09e716d None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:07.420164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 685/1350] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:04:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:07.426031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738389f9-a551-41ca-ab95-6779524d2c25 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:07.426984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738389f9-a551-41ca-ab95-6779524d2c25 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:07.427415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738389f9-a551-41ca-ab95-6779524d2c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:07.526332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-738389f9-a551-41ca-ab95-6779524d2c25 None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:07.527887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-738389f9-a551-41ca-ab95-6779524d2c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:07.697905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 666/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:04:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:11.889053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a260f044-e4aa-4836-a769-78242707fcc2 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:11.889561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a260f044-e4aa-4836-a769-78242707fcc2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:11.889561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a260f044-e4aa-4836-a769-78242707fcc2 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:11.891096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 686/1352] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:11.896800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cb21bf-c638-4cb2-8fc2-c495560152e8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:11.897254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cb21bf-c638-4cb2-8fc2-c495560152e8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:11.897653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cb21bf-c638-4cb2-8fc2-c495560152e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:11.921017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=59 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:11.954288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:11.978757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a8cb21bf-c638-4cb2-8fc2-c495560152e8 None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:11.979985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a8cb21bf-c638-4cb2-8fc2-c495560152e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:12.093447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 667/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:04:11 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:13.336440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50fa00-ff9b-4666-8df2-28ce31179858 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:13.337005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50fa00-ff9b-4666-8df2-28ce31179858 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:13.337005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a50fa00-ff9b-4666-8df2-28ce31179858 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:13.338420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 687/1354] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:04:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:13.343573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88589232-205e-4015-ad94-d9a435783499 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:13.344002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88589232-205e-4015-ad94-d9a435783499 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:13.344242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88589232-205e-4015-ad94-d9a435783499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:13.420683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-88589232-205e-4015-ad94-d9a435783499 None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:13.421906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-88589232-205e-4015-ad94-d9a435783499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:13.463518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:13.530330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 668/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:04:13 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:17.136390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c7645-4d93-4ee9-b996-b26e7b960f27 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:17.136923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c7645-4d93-4ee9-b996-b26e7b960f27 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:17.136923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c7645-4d93-4ee9-b996-b26e7b960f27 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:17.138224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 688/1356] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:17.143554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee6afdf-69df-467b-a4e9-6bd592590ff9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:17.144321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee6afdf-69df-467b-a4e9-6bd592590ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:17.144601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee6afdf-69df-467b-a4e9-6bd592590ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:17.218693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-eee6afdf-69df-467b-a4e9-6bd592590ff9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:17.219806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eee6afdf-69df-467b-a4e9-6bd592590ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:17.319798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 669/1357] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:17.642144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02daaed3-fa7f-4130-8982-fda0a22864c1 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:17.642428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02daaed3-fa7f-4130-8982-fda0a22864c1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:17.642935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02daaed3-fa7f-4130-8982-fda0a22864c1 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:17.644444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 689/1358] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 29 02:04:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:17.668488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce91288-aa98-4354-8f15-7186a7f0c6c4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:17.668849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce91288-aa98-4354-8f15-7186a7f0c6c4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:17.669327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce91288-aa98-4354-8f15-7186a7f0c6c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:17.688085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:17.757422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8ce91288-aa98-4354-8f15-7186a7f0c6c4 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:17.758624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8ce91288-aa98-4354-8f15-7186a7f0c6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:17.885344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 670/1359] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 29 02:04:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:17.893944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02daaed3-fa7f-4130-8982-fda0a22864c1 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:17.895469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-02daaed3-fa7f-4130-8982-fda0a22864c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:18.050210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02daaed3-fa7f-4130-8982-fda0a22864c1 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:18.126808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:18.174969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:18.175593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:18.175845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:18.176024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:18.177628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:18.279284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:18.281057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afff711-6399-4318-8110-52da7be1fbae service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:18.299926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 690/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:20.675626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636ea3a8-4e03-457b-906d-a911ad3471e2 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:20.675626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636ea3a8-4e03-457b-906d-a911ad3471e2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:20.676431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636ea3a8-4e03-457b-906d-a911ad3471e2 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:20.677439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 671/1361] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:20.682694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa66cc2-cfcb-4322-a239-3c47c7819b94 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:20.682694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa66cc2-cfcb-4322-a239-3c47c7819b94 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:20.683453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa66cc2-cfcb-4322-a239-3c47c7819b94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:20.750958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-caa66cc2-cfcb-4322-a239-3c47c7819b94 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:20.752705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-caa66cc2-cfcb-4322-a239-3c47c7819b94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:20.793639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:20.857014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 691/1362] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:21.143546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636ea3a8-4e03-457b-906d-a911ad3471e2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:21.145177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-636ea3a8-4e03-457b-906d-a911ad3471e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:21.214127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:21.320349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636ea3a8-4e03-457b-906d-a911ad3471e2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:21.453895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:21.454505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:21.454786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:21.455624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:21.457241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:21.480859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:21.550850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:21.552444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c521d1-f113-4fe1-b07f-09074bb87fee service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:21.573180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 672/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:24.068123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f05f2b-76e4-4f49-88ec-487831ee1edc None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:24.068740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f05f2b-76e4-4f49-88ec-487831ee1edc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:24.068981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f05f2b-76e4-4f49-88ec-487831ee1edc None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:24.070843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 692/1364] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:24.075842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98425b69-b18f-4b24-b874-97d1b863262c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:24.076274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98425b69-b18f-4b24-b874-97d1b863262c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:24.076451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98425b69-b18f-4b24-b874-97d1b863262c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:24.143850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-98425b69-b18f-4b24-b874-97d1b863262c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:24.145085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-98425b69-b18f-4b24-b874-97d1b863262c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:24.248382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:24.261199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 673/1365] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:24.537221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a769ae1e-6ea2-482a-92d8-9d30084b7915 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:24.537503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a769ae1e-6ea2-482a-92d8-9d30084b7915 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:24.537961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a769ae1e-6ea2-482a-92d8-9d30084b7915 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:24.539280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 693/1366] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 29 02:04:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:24.545529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65448fe1-ced4-4591-89c8-3a5dbffe40f4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:24.545886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65448fe1-ced4-4591-89c8-3a5dbffe40f4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:24.546236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65448fe1-ced4-4591-89c8-3a5dbffe40f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:24.633167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-65448fe1-ced4-4591-89c8-3a5dbffe40f4 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:24.634356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-65448fe1-ced4-4591-89c8-3a5dbffe40f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:24.754817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 674/1367] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 29 02:04:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:24.763181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a769ae1e-6ea2-482a-92d8-9d30084b7915 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:24.764738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a769ae1e-6ea2-482a-92d8-9d30084b7915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:24.921494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a769ae1e-6ea2-482a-92d8-9d30084b7915 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:24.944985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:25.042416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:25.043186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:25.043566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:25.043955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:25.045715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:25.140403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:25.142097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f137f8ae-2188-4b76-8810-01801593952f service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:25.161165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 694/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:25.311647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a23da1d-0b0e-4eef-af2f-d85cfde07f01 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:25.312234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a23da1d-0b0e-4eef-af2f-d85cfde07f01 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:25.312234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a23da1d-0b0e-4eef-af2f-d85cfde07f01 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:25.313762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 675/1369] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:25.319747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf17ee1-9783-423e-be3a-07537dfe51ee None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:25.320038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf17ee1-9783-423e-be3a-07537dfe51ee None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:25.320324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf17ee1-9783-423e-be3a-07537dfe51ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:25.357868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:25.401637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-daf17ee1-9783-423e-be3a-07537dfe51ee None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:25.402809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-daf17ee1-9783-423e-be3a-07537dfe51ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:25.518549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 695/1370] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:25.856748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a91106-c718-4ee1-8734-ae9c72fa1cf5 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:25.856748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a91106-c718-4ee1-8734-ae9c72fa1cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:25.856748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a91106-c718-4ee1-8734-ae9c72fa1cf5 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:25.856748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 676/1371] 10.222.0.27 () {54 vars in 963 bytes} [Thu Sep 29 02:04:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:25.860763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb4fc68-6ed4-4f0b-ad8e-bbac6b06b1c5 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:25.861681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb4fc68-6ed4-4f0b-ad8e-bbac6b06b1c5 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:25.861945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb4fc68-6ed4-4f0b-ad8e-bbac6b06b1c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:25.941729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8fb4fc68-6ed4-4f0b-ad8e-bbac6b06b1c5 None None] MFA Rules not processed for user `09c034ad1ccd45e78a4b3aeab2129227`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:25.942946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8fb4fc68-6ed4-4f0b-ad8e-bbac6b06b1c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:25.994816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:26.026716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 696/1372] 10.222.0.27 () {58 vars in 1060 bytes} [Thu Sep 29 02:04:25 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:26.035125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a91106-c718-4ee1-8734-ae9c72fa1cf5 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:26.036715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b8a91106-c718-4ee1-8734-ae9c72fa1cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:26.091187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:26.149589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a91106-c718-4ee1-8734-ae9c72fa1cf5 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:26.233464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:26.234074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:26.234350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:26.234628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:26.237038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:26.353643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:26.370569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:26.372070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e45c69-be49-4e8f-aaac-0bde83bd94be service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:26.393599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 677/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:27.404894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b89a6-aa1d-4bf1-bca9-3405c8cc2b30 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:27.405412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b89a6-aa1d-4bf1-bca9-3405c8cc2b30 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:27.405412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b89a6-aa1d-4bf1-bca9-3405c8cc2b30 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:27.406720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 697/1374] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:27.412556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ee196-2f59-4240-94b4-b565711de56d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:27.412985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ee196-2f59-4240-94b4-b565711de56d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:27.413149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ee196-2f59-4240-94b4-b565711de56d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:27.478030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-da2ee196-2f59-4240-94b4-b565711de56d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:27.479245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da2ee196-2f59-4240-94b4-b565711de56d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:27.576501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 678/1375] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:27.875160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b89a6-aa1d-4bf1-bca9-3405c8cc2b30 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:27.877332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b45b89a6-aa1d-4bf1-bca9-3405c8cc2b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:28.036414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b89a6-aa1d-4bf1-bca9-3405c8cc2b30 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:28.089058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:28.151171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:28.151785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:28.152062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:28.152333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:28.154078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:28.247746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:28.249175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c1cf88-b9ec-4a72-96e3-c86a84145f7d service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:28.272057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 698/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:30.489336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43442271-3b3d-4a6d-b4fb-65169fc18b7e None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:30.489843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43442271-3b3d-4a6d-b4fb-65169fc18b7e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:30.490092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43442271-3b3d-4a6d-b4fb-65169fc18b7e None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:30.492728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 679/1377] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:30.499057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8984d7a-6e5b-4083-a75e-80d656f0c14c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:30.499530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8984d7a-6e5b-4083-a75e-80d656f0c14c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:30.499636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8984d7a-6e5b-4083-a75e-80d656f0c14c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:30.571985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f8984d7a-6e5b-4083-a75e-80d656f0c14c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:30.573445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f8984d7a-6e5b-4083-a75e-80d656f0c14c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:30.581051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:30.685160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 699/1378] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:30.980410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43442271-3b3d-4a6d-b4fb-65169fc18b7e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:30.982163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-43442271-3b3d-4a6d-b4fb-65169fc18b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:31.045466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:31.133976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43442271-3b3d-4a6d-b4fb-65169fc18b7e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:31.260759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:31.261542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:31.261926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:31.262300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:31.263986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:31.284102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:31.360468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:31.362789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a94272b-0ff4-4790-9ffd-cd78adae5646 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:31.382072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 680/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:33.599270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e24746-bafc-40b3-9589-814b2218172a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:33.599813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e24746-bafc-40b3-9589-814b2218172a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:33.599813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e24746-bafc-40b3-9589-814b2218172a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:33.601308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 700/1380] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:04:33.606349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b48f01-d1b4-43bb-b1fb-bae7a65b9756 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:33.606767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b48f01-d1b4-43bb-b1fb-bae7a65b9756 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:33.606936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b48f01-d1b4-43bb-b1fb-bae7a65b9756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:33.676090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-69b48f01-d1b4-43bb-b1fb-bae7a65b9756 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:33.677341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-69b48f01-d1b4-43bb-b1fb-bae7a65b9756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:33.788092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:33.798937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 681/1381] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:34.118896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e24746-bafc-40b3-9589-814b2218172a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:34.120457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-98e24746-bafc-40b3-9589-814b2218172a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:34.271327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:34.282319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e24746-bafc-40b3-9589-814b2218172a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:34.425804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:34.426527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:34.426870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:34.427210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:34.429332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:34.523329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:34.524992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1348237-bd27-412d-8bed-b70e4312c87a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:34.535501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:34.550622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 701/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:36.773663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d64b0b-9d1e-42be-9fae-1285f850d5b0 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:36.774199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d64b0b-9d1e-42be-9fae-1285f850d5b0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:36.774199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d64b0b-9d1e-42be-9fae-1285f850d5b0 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:36.775547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 682/1383] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:04:36.780951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64190915-702a-43d0-9da6-dcba2744a2fa None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:36.781528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64190915-702a-43d0-9da6-dcba2744a2fa None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:36.781528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64190915-702a-43d0-9da6-dcba2744a2fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:36.856689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-64190915-702a-43d0-9da6-dcba2744a2fa None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:36.858261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-64190915-702a-43d0-9da6-dcba2744a2fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:36.959940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 702/1384] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:37.255456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d64b0b-9d1e-42be-9fae-1285f850d5b0 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:37.257034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-42d64b0b-9d1e-42be-9fae-1285f850d5b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:37.440256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d64b0b-9d1e-42be-9fae-1285f850d5b0 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:37.516481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:37.572378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:37.573019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:37.573351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:37.573625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:37.575250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:37.671188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:37.672804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bd174a-d52e-4b4d-89fc-f3af32efad7b service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:37.691859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 683/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:40.068211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92520d3-8026-4398-94d2-43500ebddf8b None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:40.068795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92520d3-8026-4398-94d2-43500ebddf8b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:40.068795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92520d3-8026-4398-94d2-43500ebddf8b None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:40.070390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 703/1386] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:40.075768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e194e56-4a54-409c-b719-854e0f962412 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:40.076232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e194e56-4a54-409c-b719-854e0f962412 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:40.076446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e194e56-4a54-409c-b719-854e0f962412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:40.141858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1e194e56-4a54-409c-b719-854e0f962412 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:40.143723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1e194e56-4a54-409c-b719-854e0f962412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:40.175963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:40.241590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 684/1387] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:40.544900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92520d3-8026-4398-94d2-43500ebddf8b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:40.548108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c92520d3-8026-4398-94d2-43500ebddf8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:40.634491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:40.705057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92520d3-8026-4398-94d2-43500ebddf8b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:40.820324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:40.820859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:40.821173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:40.821444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:40.823059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:40.868729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:40.920401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:40.922031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2afeb2-1250-413e-a46d-96437f42f2d2 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:40.940668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 704/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:41.740797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fdf0cb-2b18-45ed-b077-f7fdaaecd51c None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:41.741337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fdf0cb-2b18-45ed-b077-f7fdaaecd51c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:41.741337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fdf0cb-2b18-45ed-b077-f7fdaaecd51c None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:41.742789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 685/1389] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:41.748028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de26888-7801-4230-a163-93c0b0bb8bfc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:41.748446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de26888-7801-4230-a163-93c0b0bb8bfc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:41.748580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de26888-7801-4230-a163-93c0b0bb8bfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:41.821486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1de26888-7801-4230-a163-93c0b0bb8bfc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:41.823015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1de26888-7801-4230-a163-93c0b0bb8bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:41.934519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 705/1390] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:42.237785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8d9f9-cb2e-4f06-9c15-437e7833b152 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:42.238164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8d9f9-cb2e-4f06-9c15-437e7833b152 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:42.238508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b8d9f9-cb2e-4f06-9c15-437e7833b152 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:42.239912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 686/1391] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 29 02:04:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:04:42.245959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db5b910-68f4-4f0b-928c-06c699b60ebc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:42.246252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db5b910-68f4-4f0b-928c-06c699b60ebc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:42.246558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db5b910-68f4-4f0b-928c-06c699b60ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:42.262368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:42.323049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0db5b910-68f4-4f0b-928c-06c699b60ebc None None] MFA Rules not processed for user `0298fb95ae9341c6a1eb8da546d93cb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:42.324257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0db5b910-68f4-4f0b-928c-06c699b60ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:42.401400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 706/1392] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 29 02:04:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:42.411140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b8d9f9-cb2e-4f06-9c15-437e7833b152 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:42.412763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f7b8d9f9-cb2e-4f06-9c15-437e7833b152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:42.521967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b8d9f9-cb2e-4f06-9c15-437e7833b152 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:42.585340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:42.602391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:42.602998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:42.603280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:42.603558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:42.605200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:42.725828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:42.727495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b9dc0e-e389-487b-842b-477919015b86 service glance] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:42.748190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 687/1393] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:43.144667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bab053-f6bb-412f-a4d2-0b74cca782dc None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:43.145259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bab053-f6bb-412f-a4d2-0b74cca782dc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:43.145259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bab053-f6bb-412f-a4d2-0b74cca782dc None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:43.146581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 707/1394] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:43.152344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:43.152667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:43.152973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:43.240406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:43.241777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:43.289183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:43.351474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 688/1395] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:43.546371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dce53-a579-4b49-aa5d-784348ceba65 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:43.546728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dce53-a579-4b49-aa5d-784348ceba65 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:43.547070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dce53-a579-4b49-aa5d-784348ceba65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:43.630270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cf2dce53-a579-4b49-aa5d-784348ceba65 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:43.631822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cf2dce53-a579-4b49-aa5d-784348ceba65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:43.659608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:43.660945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:43.702521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:43.742238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 708/1396] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:04:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:43.789780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6090ef7-dcfb-4c5c-a9ab-2e9b40e87c59 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:43.791130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f321b31-a65f-4e9f-9b8a-c0d6fac1d7ab None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:43.791543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f321b31-a65f-4e9f-9b8a-c0d6fac1d7ab None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:43.791936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f321b31-a65f-4e9f-9b8a-c0d6fac1d7ab None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:43.793666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 709/1397] 10.222.0.27 () {54 vars in 943 bytes} [Thu Sep 29 02:04:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:43.801006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:43.801338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:43.801615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:43.846971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:43.881434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] MFA Rules not processed for user `10649368308f4dc4b8b887319cbcac8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:43.882647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:43.910431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:43.911036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:43.911312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:43.911581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:43.913197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:43.992086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 710/1398] 10.222.0.27 () {58 vars in 1040 bytes} [Thu Sep 29 02:04:43 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:44.001414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:44.011118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92db1de5-ae40-4a69-85dc-77e6e6278f4c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:44.011879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:44.014095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e343d1-4cfc-4f70-9aba-92c3cbcb8709 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:44.037373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 689/1399] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:04:44.037996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:44.128366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:44.129036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:44.129316 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:44.129588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:44.131453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:44.228775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:44.230436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be3d604-0a0f-46ef-bee8-7eba18846b2a service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:44.239743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 711/1400] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:04:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 29 02:04:46.245876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda1d078-c7d6-4763-b3be-4df1f63e3fdd None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:46.246436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda1d078-c7d6-4763-b3be-4df1f63e3fdd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:46.246649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda1d078-c7d6-4763-b3be-4df1f63e3fdd None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:46.248074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 690/1401] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:46.253536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac7621-f861-4e22-9346-ca501c167c74 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:46.254436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac7621-f861-4e22-9346-ca501c167c74 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:46.254735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac7621-f861-4e22-9346-ca501c167c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:46.328717 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fbac7621-f861-4e22-9346-ca501c167c74 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:46.330839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fbac7621-f861-4e22-9346-ca501c167c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:46.338778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:46.452912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 712/1402] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:46.741932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda1d078-c7d6-4763-b3be-4df1f63e3fdd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:46.743495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cda1d078-c7d6-4763-b3be-4df1f63e3fdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:46.820373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:46.913406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda1d078-c7d6-4763-b3be-4df1f63e3fdd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:47.038399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:47.039018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:47.039299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:47.039575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:47.041902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:47.075785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:47.143781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:47.145565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2256af-e0e3-4a1e-9c0f-98f65613efc6 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:47.164656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 691/1403] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:49.445773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab41475-0a82-444c-ae7c-036ddd20c20a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:49.446351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab41475-0a82-444c-ae7c-036ddd20c20a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:49.446351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab41475-0a82-444c-ae7c-036ddd20c20a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:49.447650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 713/1404] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:49.452892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c099fe-6434-447e-abbd-4c30561c1d08 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:49.453335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c099fe-6434-447e-abbd-4c30561c1d08 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:49.453443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c099fe-6434-447e-abbd-4c30561c1d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:49.527025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c7c099fe-6434-447e-abbd-4c30561c1d08 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:49.528250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c7c099fe-6434-447e-abbd-4c30561c1d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:49.642207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 692/1405] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:49.934069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab41475-0a82-444c-ae7c-036ddd20c20a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:49.935616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5ab41475-0a82-444c-ae7c-036ddd20c20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:50.091036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:50.103667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab41475-0a82-444c-ae7c-036ddd20c20a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:50.228832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:50.229495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:50.229775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:50.230044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:50.231692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:50.327283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:50.328947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f9463b-7c98-498a-9367-a40a2823fbf8 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:50.339131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:50.349551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 714/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:52.560433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e44dc-8838-40a4-8f08-4cd19c64cd11 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:52.561385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e44dc-8838-40a4-8f08-4cd19c64cd11 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:52.561667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e44dc-8838-40a4-8f08-4cd19c64cd11 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:52.563110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 693/1407] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:04:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:52.568741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e05660-8a50-46d6-843e-76f446db6486 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:52.569500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e05660-8a50-46d6-843e-76f446db6486 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:52.569606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e05660-8a50-46d6-843e-76f446db6486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:52.648818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-66e05660-8a50-46d6-843e-76f446db6486 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:52.650234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-66e05660-8a50-46d6-843e-76f446db6486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:52.760106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 715/1408] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:53.051401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6e44dc-8838-40a4-8f08-4cd19c64cd11 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:53.053181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5c6e44dc-8838-40a4-8f08-4cd19c64cd11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:53.056157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:53.219360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6e44dc-8838-40a4-8f08-4cd19c64cd11 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:53.300748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:53.345761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:53.346483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:53.346844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:53.347187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:53.348856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:53.442512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:53.444491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b40338-f29c-4481-80f3-06cc22f8ade8 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:53.465096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 694/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:02.771700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=67 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:03.461635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=61 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:07.425683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fe7067-56dd-4963-9102-89922d3ea0ea None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:07.426240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fe7067-56dd-4963-9102-89922d3ea0ea None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:07.426240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fe7067-56dd-4963-9102-89922d3ea0ea None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:07.427555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 716/1410] 10.222.0.27 () {54 vars in 918 bytes} [Thu Sep 29 02:05:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:07.432541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed9d13e-ec15-4c7c-a144-f1cc2fa4ebc3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:07.432989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed9d13e-ec15-4c7c-a144-f1cc2fa4ebc3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:07.433119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed9d13e-ec15-4c7c-a144-f1cc2fa4ebc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:07.507293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1ed9d13e-ec15-4c7c-a144-f1cc2fa4ebc3 None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:07.508505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1ed9d13e-ec15-4c7c-a144-f1cc2fa4ebc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:07.613176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 695/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Sep 29 02:05:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:07.627432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20414718-9b10-43a9-bdb3-7a31a86ae933 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:07.627715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20414718-9b10-43a9-bdb3-7a31a86ae933 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:07.627988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20414718-9b10-43a9-bdb3-7a31a86ae933 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:07.629342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 717/1412] 10.222.0.27 () {54 vars in 918 bytes} [Thu Sep 29 02:05:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:07.634088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544abbd6-c71f-4b54-a0d3-80e236c68c0b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:07.634372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544abbd6-c71f-4b54-a0d3-80e236c68c0b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:07.634640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544abbd6-c71f-4b54-a0d3-80e236c68c0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:07.708386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:07.711394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-544abbd6-c71f-4b54-a0d3-80e236c68c0b None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:07.713162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-544abbd6-c71f-4b54-a0d3-80e236c68c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:07.825039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 696/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Sep 29 02:05:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:10.375480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880d5e88-7e47-44bd-9571-9bf54aa551f0 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:10.376040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880d5e88-7e47-44bd-9571-9bf54aa551f0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:10.376040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880d5e88-7e47-44bd-9571-9bf54aa551f0 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:10.377395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 718/1414] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:10.382546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a0039c-8bf8-429b-a571-5bdc79cbedc4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:10.382971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a0039c-8bf8-429b-a571-5bdc79cbedc4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:10.383082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a0039c-8bf8-429b-a571-5bdc79cbedc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:10.447229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b1a0039c-8bf8-429b-a571-5bdc79cbedc4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:10.448417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b1a0039c-8bf8-429b-a571-5bdc79cbedc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:10.501159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:10.541667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 697/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:10.827189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880d5e88-7e47-44bd-9571-9bf54aa551f0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:10.828752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-880d5e88-7e47-44bd-9571-9bf54aa551f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:10.972629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880d5e88-7e47-44bd-9571-9bf54aa551f0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:11.041673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:11.081502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:11.082029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:11.082305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:11.082571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:11.084133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:11.169571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:11.171144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ef61a-ef6d-476f-ab57-765634dbff1a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:11.188877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 719/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:12.339435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabbc555-ed98-4902-a9f8-607889073dec None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:12.340025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabbc555-ed98-4902-a9f8-607889073dec None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:12.340025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabbc555-ed98-4902-a9f8-607889073dec None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:12.341337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 698/1417] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:12.347528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2c7fb-dd90-426d-b14e-6c714e432747 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:12.347994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2c7fb-dd90-426d-b14e-6c714e432747 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:12.348084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2c7fb-dd90-426d-b14e-6c714e432747 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:12.422361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2ae2c7fb-dd90-426d-b14e-6c714e432747 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:12.423557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2ae2c7fb-dd90-426d-b14e-6c714e432747 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:12.450361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:12.518513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 720/1418] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:12.682140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabbc555-ed98-4902-a9f8-607889073dec None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:12.683652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dabbc555-ed98-4902-a9f8-607889073dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:12.826005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabbc555-ed98-4902-a9f8-607889073dec None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:12.865993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:12.936842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:12.938040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:12.938318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:12.938589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:12.940189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:13.027264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:13.028851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785b166-d744-418f-8655-de3f431009b8 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:13.046968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 699/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:14.178245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8180b44-f527-4d4d-a077-e8df638e3e45 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:14.178793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8180b44-f527-4d4d-a077-e8df638e3e45 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:14.178793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8180b44-f527-4d4d-a077-e8df638e3e45 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:14.180122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 721/1420] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:14.186175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e355713f-b642-4835-ac75-7313528af7bb None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:14.186624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e355713f-b642-4835-ac75-7313528af7bb None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:14.186736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e355713f-b642-4835-ac75-7313528af7bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:14.240601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:14.262695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e355713f-b642-4835-ac75-7313528af7bb None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:14.263890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e355713f-b642-4835-ac75-7313528af7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:14.358087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 700/1421] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:14.507826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8180b44-f527-4d4d-a077-e8df638e3e45 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:14.509379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b8180b44-f527-4d4d-a077-e8df638e3e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:14.655230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8180b44-f527-4d4d-a077-e8df638e3e45 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:14.667503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:14.768749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:14.769295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:14.769568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:14.769831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:14.771436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:14.857779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:14.859430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf06640-d9d5-4667-897f-e1d6a52f9455 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:14.877187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 722/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:15.995459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d53c1d9-5b2c-4b8b-850b-78b648e519cb None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:15.995994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d53c1d9-5b2c-4b8b-850b-78b648e519cb None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:15.995994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d53c1d9-5b2c-4b8b-850b-78b648e519cb None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:15.997437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 701/1423] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:16.004129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b18184-5a44-4c98-8ad9-36e12d6420a2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:16.004561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b18184-5a44-4c98-8ad9-36e12d6420a2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:16.004701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b18184-5a44-4c98-8ad9-36e12d6420a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:16.025610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:16.080628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-07b18184-5a44-4c98-8ad9-36e12d6420a2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:16.081851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-07b18184-5a44-4c98-8ad9-36e12d6420a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:16.175871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 723/1424] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:16.327442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d53c1d9-5b2c-4b8b-850b-78b648e519cb None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:16.329423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0d53c1d9-5b2c-4b8b-850b-78b648e519cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:16.436382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:16.472124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d53c1d9-5b2c-4b8b-850b-78b648e519cb None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:16.581337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:16.581892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:16.582164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:16.582431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:16.584015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:16.661774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:16.675944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:16.678135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1aa8e4-6411-4ed6-bce4-655030727d96 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:16.697064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 702/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:17.816578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68307f8-d7c6-4b6b-9a78-c47206283008 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:17.817172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68307f8-d7c6-4b6b-9a78-c47206283008 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:17.817172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68307f8-d7c6-4b6b-9a78-c47206283008 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:17.818539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 724/1426] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:17.825048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2259bc5-a6f2-4620-aa67-a5a795e523d6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:17.825480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2259bc5-a6f2-4620-aa67-a5a795e523d6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:17.825599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2259bc5-a6f2-4620-aa67-a5a795e523d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:17.901324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e2259bc5-a6f2-4620-aa67-a5a795e523d6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:17.902451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e2259bc5-a6f2-4620-aa67-a5a795e523d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:18.000785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 703/1427] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:18.149093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68307f8-d7c6-4b6b-9a78-c47206283008 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:18.150588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b68307f8-d7c6-4b6b-9a78-c47206283008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:18.231426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:18.291576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68307f8-d7c6-4b6b-9a78-c47206283008 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:18.399870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:18.400472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:18.400743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:18.401042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:18.402596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:18.449661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:18.489317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:18.490868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecf504a-54a1-457d-b30f-25822d19fcd9 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:18.508532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 725/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:19.583044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7ebc2c-d1cb-42f2-91cb-fd24ee43f149 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:19.583605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7ebc2c-d1cb-42f2-91cb-fd24ee43f149 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:19.583605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7ebc2c-d1cb-42f2-91cb-fd24ee43f149 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:19.584960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 704/1429] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:19.591363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ea2bdd-cd88-4a4a-ab34-0d4709866fb0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:19.591775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ea2bdd-cd88-4a4a-ab34-0d4709866fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:19.591926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ea2bdd-cd88-4a4a-ab34-0d4709866fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:19.666568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-45ea2bdd-cd88-4a4a-ab34-0d4709866fb0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:19.667776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-45ea2bdd-cd88-4a4a-ab34-0d4709866fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:19.762410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 726/1430] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:19.911837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7ebc2c-d1cb-42f2-91cb-fd24ee43f149 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:19.913328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-af7ebc2c-d1cb-42f2-91cb-fd24ee43f149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:19.961548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:20.054859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7ebc2c-d1cb-42f2-91cb-fd24ee43f149 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:20.165329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:20.165849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:20.166123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:20.166387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:20.167988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:20.175253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:20.255301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:20.256917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f77fa-3749-413b-8af4-3be68382ed5d service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:20.274335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 705/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:21.328620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de5ed5a-7525-4aeb-998c-7922b27dd929 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:21.329214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de5ed5a-7525-4aeb-998c-7922b27dd929 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:21.329214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de5ed5a-7525-4aeb-998c-7922b27dd929 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:21.330562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 727/1432] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:21 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:21.336930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fea272-071f-4126-97bb-1799d35423f3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:21.337379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fea272-071f-4126-97bb-1799d35423f3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:21.337474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fea272-071f-4126-97bb-1799d35423f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:21.409922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-80fea272-071f-4126-97bb-1799d35423f3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:21.411112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-80fea272-071f-4126-97bb-1799d35423f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:21.477421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:21.506153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 706/1433] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:21.656327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de5ed5a-7525-4aeb-998c-7922b27dd929 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:21.657913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9de5ed5a-7525-4aeb-998c-7922b27dd929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:21.675787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:21.801236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de5ed5a-7525-4aeb-998c-7922b27dd929 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:21.886369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:21.912567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:21.913203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:21.913487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:21.913759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:21.915345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:22.004201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:22.005847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62924503-e8f2-4b20-b39d-6dee4cb8024a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:22.023193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 728/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:23.063243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906f4cd-6161-4c76-b03e-b60bc6e5141f None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:23.063802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906f4cd-6161-4c76-b03e-b60bc6e5141f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:23.063802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906f4cd-6161-4c76-b03e-b60bc6e5141f None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:23.065199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 707/1435] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:05:23.071724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ebf8d9-7363-4ad7-bed1-dc39145ff2d2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:23.072157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ebf8d9-7363-4ad7-bed1-dc39145ff2d2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:23.072266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ebf8d9-7363-4ad7-bed1-dc39145ff2d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:23.149803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-92ebf8d9-7363-4ad7-bed1-dc39145ff2d2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:23.151015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-92ebf8d9-7363-4ad7-bed1-dc39145ff2d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:23.191665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:23.247419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 729/1436] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:23.395712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8906f4cd-6161-4c76-b03e-b60bc6e5141f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:23.397182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8906f4cd-6161-4c76-b03e-b60bc6e5141f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:23.545574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8906f4cd-6161-4c76-b03e-b60bc6e5141f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:23.601730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:23.659661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:23.660189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:23.660470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:23.660744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:23.662374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:23.750322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:23.751974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277da263-65b4-4d2c-af29-ddb33567579c service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:23.771511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 708/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:24.813182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf67d5-6940-4df4-915d-0875db0886f5 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:24.813766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf67d5-6940-4df4-915d-0875db0886f5 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:24.813766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf67d5-6940-4df4-915d-0875db0886f5 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:24.815071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 730/1438] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:05:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:24.821461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb167a72-aef5-4ad2-bc61-dadfde59f797 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:24.821876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb167a72-aef5-4ad2-bc61-dadfde59f797 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:24.822010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb167a72-aef5-4ad2-bc61-dadfde59f797 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:24.896545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-fb167a72-aef5-4ad2-bc61-dadfde59f797 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:24.897715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fb167a72-aef5-4ad2-bc61-dadfde59f797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:24.899715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:24.992676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 709/1439] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:05:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:25.141868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf67d5-6940-4df4-915d-0875db0886f5 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:25.143386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1ebf67d5-6940-4df4-915d-0875db0886f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:25.285572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf67d5-6940-4df4-915d-0875db0886f5 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:25.312395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:25.394854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:25.395452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:25.395725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:25.395990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:25.397598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:25.486075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:25.487714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41bebf8-80c1-4d22-a315-4d5993e4e455 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:25.504993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 731/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:05:27.578735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccf95ed-1537-4444-b095-f0e59e012e76 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:27.579273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccf95ed-1537-4444-b095-f0e59e012e76 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:27.579273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccf95ed-1537-4444-b095-f0e59e012e76 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:27.580639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 710/1441] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:27.586027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003c7509-678d-486d-a3ec-d328cf371f4c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:27.586479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003c7509-678d-486d-a3ec-d328cf371f4c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:27.586584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003c7509-678d-486d-a3ec-d328cf371f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:27.627212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:27.655987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-003c7509-678d-486d-a3ec-d328cf371f4c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:27.657229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-003c7509-678d-486d-a3ec-d328cf371f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:27.751539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 732/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:28.040857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccf95ed-1537-4444-b095-f0e59e012e76 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:28.042416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fccf95ed-1537-4444-b095-f0e59e012e76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:28.144452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccf95ed-1537-4444-b095-f0e59e012e76 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:28.176080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:28.217159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:28.218380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:28.218671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:28.218946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:28.220731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:28.331424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:28.332784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fce1a38-24f1-4348-9e0a-ed858de17728 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:28.350729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 711/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:30.631434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5112ea-786c-42fe-9ed4-266264760193 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:30.631945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5112ea-786c-42fe-9ed4-266264760193 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:30.632053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5112ea-786c-42fe-9ed4-266264760193 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:30.633473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 733/1444] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:30.638889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d5e5e-c51e-4c94-9c21-923c494fe352 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:30.639342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d5e5e-c51e-4c94-9c21-923c494fe352 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:30.639462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d5e5e-c51e-4c94-9c21-923c494fe352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:30.705484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4d4d5e5e-c51e-4c94-9c21-923c494fe352 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:30.706692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4d4d5e5e-c51e-4c94-9c21-923c494fe352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:30.719686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:30.805124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 712/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:31.089469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5112ea-786c-42fe-9ed4-266264760193 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:31.091001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7e5112ea-786c-42fe-9ed4-266264760193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:31.189613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5112ea-786c-42fe-9ed4-266264760193 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:31.202603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:31.263589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:31.264745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:31.265063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:31.265337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:31.266977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:31.373311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:31.375199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fbd956-c790-4fdf-a7ac-13c51f4dc1db service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:31.393660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 734/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:33.843063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456a262b-d827-4dae-ba44-eb3f75895d1b None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:33.843606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456a262b-d827-4dae-ba44-eb3f75895d1b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:33.843606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456a262b-d827-4dae-ba44-eb3f75895d1b None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:33.844847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 713/1447] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:33.850086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9734b25-8a2b-4526-9b4d-2a3ba4191234 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:33.850529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9734b25-8a2b-4526-9b4d-2a3ba4191234 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:33.850632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9734b25-8a2b-4526-9b4d-2a3ba4191234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:33.897776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:33.916359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a9734b25-8a2b-4526-9b4d-2a3ba4191234 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:33.917599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a9734b25-8a2b-4526-9b4d-2a3ba4191234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:34.012044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 735/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:34.296253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456a262b-d827-4dae-ba44-eb3f75895d1b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:34.297870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-456a262b-d827-4dae-ba44-eb3f75895d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:34.402745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456a262b-d827-4dae-ba44-eb3f75895d1b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:34.445438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:34.479608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:34.480219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:34.480488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:34.480768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:34.482388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:34.588662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:34.590146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d98a03-758b-40b4-a29b-e54ebc291ef6 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:34.608813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 714/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:36.866237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90526e8a-b886-4be4-8796-6f6d09d15d9a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:36.866762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90526e8a-b886-4be4-8796-6f6d09d15d9a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:36.866762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90526e8a-b886-4be4-8796-6f6d09d15d9a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:36.868021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 736/1450] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:36.873252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569a2e3b-5b26-4974-b13d-9d60cb141941 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:36.873676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569a2e3b-5b26-4974-b13d-9d60cb141941 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:36.873874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569a2e3b-5b26-4974-b13d-9d60cb141941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:36.939019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-569a2e3b-5b26-4974-b13d-9d60cb141941 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:36.940221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-569a2e3b-5b26-4974-b13d-9d60cb141941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:36.960919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:37.035744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 715/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:37.315203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90526e8a-b886-4be4-8796-6f6d09d15d9a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:37.317308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-90526e8a-b886-4be4-8796-6f6d09d15d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:37.419276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90526e8a-b886-4be4-8796-6f6d09d15d9a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:37.439882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:37.491992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:37.492585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:37.492857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:37.493150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:37.494716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:37.599876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:37.601373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbe57ad-6c2a-4458-823a-cf5a839d96aa service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:37.618897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 737/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:39.851454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f9821f-a1b9-4a48-8bdf-1c21e94755d8 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:39.852001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f9821f-a1b9-4a48-8bdf-1c21e94755d8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:39.852001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f9821f-a1b9-4a48-8bdf-1c21e94755d8 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:39.853304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 716/1453] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:39.859266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b071b-3b98-4c97-8056-1899497ee364 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:39.859712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b071b-3b98-4c97-8056-1899497ee364 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:39.859809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b071b-3b98-4c97-8056-1899497ee364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:39.921175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:39.924060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6d9b071b-3b98-4c97-8056-1899497ee364 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:39.925796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6d9b071b-3b98-4c97-8056-1899497ee364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:40.020770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 738/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:40.307121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f9821f-a1b9-4a48-8bdf-1c21e94755d8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:40.308647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-70f9821f-a1b9-4a48-8bdf-1c21e94755d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:40.409702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f9821f-a1b9-4a48-8bdf-1c21e94755d8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:40.459132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:40.482851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:40.483453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:40.483732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:40.483994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:40.485604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:40.591703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:40.593257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4e50e4-421a-4ea7-97de-a01c7cd1f97d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:40.611340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 717/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:42.787191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86681bc-a783-4ef6-9759-28be6f089650 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:42.787712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86681bc-a783-4ef6-9759-28be6f089650 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:42.787712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86681bc-a783-4ef6-9759-28be6f089650 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:42.789116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 739/1456] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:42.794562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9967e0-8321-4b88-8cbe-9fb6a4fb8a71 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:42.794999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9967e0-8321-4b88-8cbe-9fb6a4fb8a71 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:42.795089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9967e0-8321-4b88-8cbe-9fb6a4fb8a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:42.860657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2e9967e0-8321-4b88-8cbe-9fb6a4fb8a71 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:42.861809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2e9967e0-8321-4b88-8cbe-9fb6a4fb8a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:42.890587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:42.956424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 718/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:43.243178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86681bc-a783-4ef6-9759-28be6f089650 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:43.244748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f86681bc-a783-4ef6-9759-28be6f089650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:43.345753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86681bc-a783-4ef6-9759-28be6f089650 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:43.373575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:43.417138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:43.417765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:43.418043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:43.418311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:43.419927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:43.526018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:43.527491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f3e2f5-31f5-486a-ba84-4d2cfaa9b210 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:43.546250 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 740/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:45.741648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b177c-5b17-41a4-8ecd-355ba9321b8e None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:45.742218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b177c-5b17-41a4-8ecd-355ba9321b8e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:45.742218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b177c-5b17-41a4-8ecd-355ba9321b8e None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:45.743577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 719/1459] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:45.749277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda19687-4875-46a9-8d20-87d8aea3cf27 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:45.749713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda19687-4875-46a9-8d20-87d8aea3cf27 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:45.749855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda19687-4875-46a9-8d20-87d8aea3cf27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:45.814615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-dda19687-4875-46a9-8d20-87d8aea3cf27 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:45.815802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-dda19687-4875-46a9-8d20-87d8aea3cf27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:45.824033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:45.910774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 741/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:46.188909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96b177c-5b17-41a4-8ecd-355ba9321b8e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:46.190427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b96b177c-5b17-41a4-8ecd-355ba9321b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:46.290117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96b177c-5b17-41a4-8ecd-355ba9321b8e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:46.344272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:46.362418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:46.363019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:46.363296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:46.363565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:46.365336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:46.471778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:46.473224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb96444-e7d7-4a91-b07f-d6ae02e6007d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:46.491287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 720/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:48.715254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da9e0ce-91d8-48b8-876f-57d966851756 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:48.715824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da9e0ce-91d8-48b8-876f-57d966851756 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:48.715824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da9e0ce-91d8-48b8-876f-57d966851756 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:48.717192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 742/1462] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:48.722538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc070f52-62b5-4db8-ab8e-f9c8757edcf4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:48.722941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc070f52-62b5-4db8-ab8e-f9c8757edcf4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:48.723093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc070f52-62b5-4db8-ab8e-f9c8757edcf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:48.789354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-fc070f52-62b5-4db8-ab8e-f9c8757edcf4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:48.790557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fc070f52-62b5-4db8-ab8e-f9c8757edcf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:48.831782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:48.893669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 721/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:49.175914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da9e0ce-91d8-48b8-876f-57d966851756 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:49.177474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7da9e0ce-91d8-48b8-876f-57d966851756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:49.279005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da9e0ce-91d8-48b8-876f-57d966851756 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:49.315237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:49.351693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:49.352295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:49.352574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:49.352843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:49.354442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:49.459369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:49.460724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76e5cc9-ad04-4332-bf5e-fd7d587d7e69 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:49.478088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 743/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:51.613982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45b0772-f626-4086-98e5-2ca1c510b376 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:51.614509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45b0772-f626-4086-98e5-2ca1c510b376 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:51.614509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45b0772-f626-4086-98e5-2ca1c510b376 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:51.615854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 722/1465] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:51.621441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769920c-4c11-4b73-b9f6-3eeb6e05187f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:51.621869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769920c-4c11-4b73-b9f6-3eeb6e05187f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:51.622012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769920c-4c11-4b73-b9f6-3eeb6e05187f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:51.686332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c769920c-4c11-4b73-b9f6-3eeb6e05187f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:51.687522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c769920c-4c11-4b73-b9f6-3eeb6e05187f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:51.705487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:51.781600 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 744/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:52.065793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45b0772-f626-4086-98e5-2ca1c510b376 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:52.067324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f45b0772-f626-4086-98e5-2ca1c510b376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:52.168915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45b0772-f626-4086-98e5-2ca1c510b376 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:52.235049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:52.242118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:52.242715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:52.242992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:52.243258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:52.244817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:52.467220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:52.468534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e63c6-0819-4cc3-8895-01bccdb709cd service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:52.485865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 723/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:54.643339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d6def-3f7e-4c72-a6af-118416b6e081 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:54.643887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d6def-3f7e-4c72-a6af-118416b6e081 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:54.643887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d6def-3f7e-4c72-a6af-118416b6e081 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:54.645253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 745/1468] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:54.650832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90b7111-56f3-4a42-85b8-a209062f9b7f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:54.651261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90b7111-56f3-4a42-85b8-a209062f9b7f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:54.651729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90b7111-56f3-4a42-85b8-a209062f9b7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:54.717942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c90b7111-56f3-4a42-85b8-a209062f9b7f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:54.719134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c90b7111-56f3-4a42-85b8-a209062f9b7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:54.765424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:54.819837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 724/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:05:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:55.104825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d6def-3f7e-4c72-a6af-118416b6e081 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:55.106378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-bd4d6def-3f7e-4c72-a6af-118416b6e081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:55.206535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d6def-3f7e-4c72-a6af-118416b6e081 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:55.254558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:55.280428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:55.281054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:55.281328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:55.281594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:55.283161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:55.387045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:55.388994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2d97d3-6ddd-44a0-b9e1-11811cd380cf service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:55.406351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 746/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:04.818244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=22 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:05.404168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=66 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:15.406670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a14cb8-d5e5-4d83-81dc-b75dd1f6c4f4 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:15.407257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a14cb8-d5e5-4d83-81dc-b75dd1f6c4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:15.407257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a14cb8-d5e5-4d83-81dc-b75dd1f6c4f4 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:15.408528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 725/1471] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:15.413881 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553f74a3-67f1-475b-beb0-5957cb3072be None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:15.414304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553f74a3-67f1-475b-beb0-5957cb3072be None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:15.414404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553f74a3-67f1-475b-beb0-5957cb3072be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:15.479966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-553f74a3-67f1-475b-beb0-5957cb3072be None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:15.481218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-553f74a3-67f1-475b-beb0-5957cb3072be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:15.575406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 747/1472] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:15.872636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e2ec00-babb-4b3b-931a-765094391ac4 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:15.872950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e2ec00-babb-4b3b-931a-765094391ac4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:15.873225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e2ec00-babb-4b3b-931a-765094391ac4 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:15.874561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 726/1473] 10.222.0.27 () {54 vars in 962 bytes} [Thu Sep 29 02:06:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:15.879712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994113a7-82a0-4eb5-9ac0-6f1663607b99 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:15.880001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994113a7-82a0-4eb5-9ac0-6f1663607b99 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:15.880276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994113a7-82a0-4eb5-9ac0-6f1663607b99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:15.955980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-994113a7-82a0-4eb5-9ac0-6f1663607b99 None None] MFA Rules not processed for user `0298fb95ae9341c6a1eb8da546d93cb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:15.957219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-994113a7-82a0-4eb5-9ac0-6f1663607b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:15.977176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:16.032475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 748/1474] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Sep 29 02:06:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:16.040111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e2ec00-babb-4b3b-931a-765094391ac4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:16.041713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-23e2ec00-babb-4b3b-931a-765094391ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:16.141804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e2ec00-babb-4b3b-931a-765094391ac4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:16.212069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:16.212663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:16.212966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:16.213239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:16.214786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:16.268671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:16.322958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:16.324383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0c42d5-4ebc-4c4f-ad5b-2f6f6d9a3f97 service glance] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:16.342268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 727/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:18.581271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f02abca-f7aa-4d5b-a69e-7ec48ce814ba None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:18.581793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f02abca-f7aa-4d5b-a69e-7ec48ce814ba None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:18.581793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f02abca-f7aa-4d5b-a69e-7ec48ce814ba None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:18.583068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 749/1476] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:18.588397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ca9b19-6115-4efc-aab6-45b7431bbabf None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:18.588879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ca9b19-6115-4efc-aab6-45b7431bbabf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:18.588994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ca9b19-6115-4efc-aab6-45b7431bbabf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:18.655437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b7ca9b19-6115-4efc-aab6-45b7431bbabf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:18.656651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b7ca9b19-6115-4efc-aab6-45b7431bbabf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:18.750975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 728/1477] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:19.029278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f02abca-f7aa-4d5b-a69e-7ec48ce814ba None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:19.030809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3f02abca-f7aa-4d5b-a69e-7ec48ce814ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:19.154009 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f02abca-f7aa-4d5b-a69e-7ec48ce814ba None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:19.201568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:19.250017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:19.250942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:19.251217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:19.251484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:19.252116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:19.253941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:06:19.253941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:06:19.253941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:06:19.255079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13cea4b-0fc8-4b5c-9ee7-093f11176dde admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:19.274527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 750/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Sep 29 02:06:19 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:06:21.442326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9689f5aa-3a58-4836-b22e-2b6635bcdb21 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:21.443164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9689f5aa-3a58-4836-b22e-2b6635bcdb21 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:21.443164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9689f5aa-3a58-4836-b22e-2b6635bcdb21 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:21.444445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 729/1479] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:21.450106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ee44ca-c68b-4ec6-85e1-adac3acff41f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:21.450502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ee44ca-c68b-4ec6-85e1-adac3acff41f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:21.450664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ee44ca-c68b-4ec6-85e1-adac3acff41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:21.514926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-97ee44ca-c68b-4ec6-85e1-adac3acff41f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:21.516132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-97ee44ca-c68b-4ec6-85e1-adac3acff41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:21.612043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 751/1480] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:21.906778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9689f5aa-3a58-4836-b22e-2b6635bcdb21 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:21.908276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9689f5aa-3a58-4836-b22e-2b6635bcdb21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:21.917887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:22.050582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9689f5aa-3a58-4836-b22e-2b6635bcdb21 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:22.128343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:22.159924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:22.160457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:22.160735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:22.161035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:22.162615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:22.250674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:22.252241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b54f51f-9c32-4924-abfa-62e198e0f7d0 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:22.269905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 730/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:24.574047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f819e29-5028-4109-bb7e-a2db0ccda573 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:24.575058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f819e29-5028-4109-bb7e-a2db0ccda573 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:24.575058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f819e29-5028-4109-bb7e-a2db0ccda573 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:24.576391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 752/1482] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:24.581714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec803e9-4b61-477e-af34-59a6e2ea75e3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:24.582158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec803e9-4b61-477e-af34-59a6e2ea75e3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:24.582276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec803e9-4b61-477e-af34-59a6e2ea75e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:24.646576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-eec803e9-4b61-477e-af34-59a6e2ea75e3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:24.647773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eec803e9-4b61-477e-af34-59a6e2ea75e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:24.693292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:24.742620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 731/1483] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:25.031393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f819e29-5028-4109-bb7e-a2db0ccda573 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:25.032951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6f819e29-5028-4109-bb7e-a2db0ccda573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:25.056044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:25.134778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f819e29-5028-4109-bb7e-a2db0ccda573 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:25.207838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:25.208429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:25.208701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:25.209086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:25.210671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:25.284289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:25.326696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:25.328145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fec6ce-dbd8-4ad3-ace5-50ccca98fc19 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:25.346293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 753/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:27.533651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea194e-ab85-4d4c-9558-b37a74f4126b None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:27.534188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea194e-ab85-4d4c-9558-b37a74f4126b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:27.534188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea194e-ab85-4d4c-9558-b37a74f4126b None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:27.535526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 732/1485] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:27.541070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf1426-a94e-4572-9179-cb7843151db7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:27.541470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf1426-a94e-4572-9179-cb7843151db7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:27.541654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf1426-a94e-4572-9179-cb7843151db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:27.607813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-b9bf1426-a94e-4572-9179-cb7843151db7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:27.609099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b9bf1426-a94e-4572-9179-cb7843151db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:27.706065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 754/1486] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:27.992609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ea194e-ab85-4d4c-9558-b37a74f4126b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:27.994205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-27ea194e-ab85-4d4c-9558-b37a74f4126b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:28.094230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ea194e-ab85-4d4c-9558-b37a74f4126b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:28.164360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:28.166018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:28.166759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:28.167209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:28.167525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:28.169441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:28.275032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:28.276448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a53785-4ad3-41a8-a3c6-ea81398d6fef service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:28.297363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 733/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:30.513811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18aa319-a579-48f8-a5cf-809f3a78fc93 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:30.514327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18aa319-a579-48f8-a5cf-809f3a78fc93 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:30.514445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18aa319-a579-48f8-a5cf-809f3a78fc93 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:30.515742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 755/1488] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:30.520884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267e3728-9a30-4552-a4f7-a550b407c47f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:30.521337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267e3728-9a30-4552-a4f7-a550b407c47f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:30.521442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267e3728-9a30-4552-a4f7-a550b407c47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:30.586471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-267e3728-9a30-4552-a4f7-a550b407c47f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:30.587271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-267e3728-9a30-4552-a4f7-a550b407c47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:30.639531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:30.681940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 734/1489] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:30.969398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18aa319-a579-48f8-a5cf-809f3a78fc93 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:30.970938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c18aa319-a579-48f8-a5cf-809f3a78fc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:31.005591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:31.077470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18aa319-a579-48f8-a5cf-809f3a78fc93 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:31.266200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:31.266796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:31.267071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:31.267336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:31.268937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:31.347145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:31.376710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:31.378174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106a7b93-6d81-427f-8e3e-eb43a9bdc351 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:31.396438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 756/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:33.745828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79cd60-97bf-48e5-bac4-67c0b3f90537 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:33.746390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79cd60-97bf-48e5-bac4-67c0b3f90537 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:33.746390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b79cd60-97bf-48e5-bac4-67c0b3f90537 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:33.747824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 735/1491] 10.222.0.27 () {54 vars in 927 bytes} [Thu Sep 29 02:06:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:06:33.753247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacc3cc-3673-402d-a742-bacf596b3a85 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:33.753647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacc3cc-3673-402d-a742-bacf596b3a85 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:33.753827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffacc3cc-3673-402d-a742-bacf596b3a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:33.818459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ffacc3cc-3673-402d-a742-bacf596b3a85 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:33.819673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ffacc3cc-3673-402d-a742-bacf596b3a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:33.912175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 757/1492] 10.222.0.27 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:34.209318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b79cd60-97bf-48e5-bac4-67c0b3f90537 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:34.210872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5b79cd60-97bf-48e5-bac4-67c0b3f90537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:34.322292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b79cd60-97bf-48e5-bac4-67c0b3f90537 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:34.395267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:34.395875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:34.396153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:34.396423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:34.398056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:34.406351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:34.504961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:34.506454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185f1edf-b2b1-4292-9a2f-1ebe351d456c service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:34.524708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 736/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:43.910694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=82 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:44.522484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=50 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:53.555849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:53.556456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:53.556562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:53.631522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:53.633268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:53.727693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 758/1494] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:07:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:53.735609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:53.736096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:53.736224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:53.808475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:53.809621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:53.904117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 737/1495] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:07:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:53.908699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:53.909934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:53.921174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d5888a-ffdb-42f7-823d-b4f7beb4e60d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:53.980530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:54.014122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:54.014722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.014998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.015264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.015893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:07:54.016190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:54.017221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:54.017221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:07:54.017221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:07:54.018104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:54.028783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 759/1496] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:07:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:07:54.036996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:54.045859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4192d-e1f1-4873-9a29-15965a3deaea None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:54.105853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:54.139107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:54.139707 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.139984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.140253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.140901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:54.142623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:54.142623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:07:54.142623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:07:54.143823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:54.197965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 738/1497] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:07:54 2022] POST /identity/v3/projects => generated 379 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:07:54.202125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:54.203630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:54.236651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81e3b99-fcbc-465d-8eaa-8ae1b2b559ce admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:54.327474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:54.328069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.328346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.328617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.329278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:54.330279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:54.330279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:07:54.330279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:07:54.331837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:54.373666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 760/1498] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:07:54 2022] POST /identity/v3/users => generated 290 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:07:54.377652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:54.379176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:54.439105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614345db-a71e-40dc-aaee-0598b5abd50e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:54.520447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:54.532208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:54.532809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.533112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.533380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.534013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:54.535017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:54.535017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:07:54.535017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:07:54.535899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:54.547800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 739/1499] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:07:54 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:07:54.551817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:54.553358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:54.564388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778299c6-d668-4cff-b025-b7cedd374830 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:54.590335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:54.658529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:54.659058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.659333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.659600 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.660239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:54.661287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:54.661287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:07:54.661287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:07:54.662167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6546cc0-dbdc-4cd5-bbcb-33de41a5b41f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:54.674581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 761/1500] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:07:54 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:07:54.678803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:54.680339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:54.714166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a39e6-8bb5-4c03-90c1-a43fe003669e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:54.805945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:54.806601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.806979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.807267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] PATH_INFO: `/v3/projects/6649e2e20a344606b65f5ebc4d5479fa/users/0226aa5f8a8e4cc29a5c70cec8b2366a/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.829121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6649e2e20a344606b65f5ebc4d5479fa, user_id=0226aa5f8a8e4cc29a5c70cec8b2366a, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:54.830144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:54.830144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:07:54.830144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:07:54.832078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7626281c-4bbd-40c4-a4fb-d3a5e8000dd5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:54.856684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 740/1501] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:07:54 2022] PUT /identity/v3/projects/6649e2e20a344606b65f5ebc4d5479fa/users/0226aa5f8a8e4cc29a5c70cec8b2366a/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:07:54.868645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec0f35a-8d74-4db2-aa84-9d0d1d0f3931 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:54.868987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec0f35a-8d74-4db2-aa84-9d0d1d0f3931 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:54.869269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec0f35a-8d74-4db2-aa84-9d0d1d0f3931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:54.925431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fec0f35a-8d74-4db2-aa84-9d0d1d0f3931 None None] MFA Rules not processed for user `0226aa5f8a8e4cc29a5c70cec8b2366a`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:54.926575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fec0f35a-8d74-4db2-aa84-9d0d1d0f3931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:55.002649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 762/1502] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:07:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:55.016155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8207df4-31fc-4f79-b727-9911a2b363fc None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:55.016443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8207df4-31fc-4f79-b727-9911a2b363fc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:55.016715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8207df4-31fc-4f79-b727-9911a2b363fc None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:55.018068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 741/1503] 10.222.0.27 () {54 vars in 963 bytes} [Thu Sep 29 02:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:55.022663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44627792-89ac-4479-a861-a6f78e37bc36 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:55.022951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44627792-89ac-4479-a861-a6f78e37bc36 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:55.023224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44627792-89ac-4479-a861-a6f78e37bc36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:55.054196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:55.098967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-44627792-89ac-4479-a861-a6f78e37bc36 None None] MFA Rules not processed for user `09c034ad1ccd45e78a4b3aeab2129227`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:55.100167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-44627792-89ac-4479-a861-a6f78e37bc36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:55.172699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 763/1504] 10.222.0.27 () {58 vars in 1060 bytes} [Thu Sep 29 02:07:55 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:55.180446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8207df4-31fc-4f79-b727-9911a2b363fc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:55.181913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c8207df4-31fc-4f79-b727-9911a2b363fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:55.230526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:55.283125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8207df4-31fc-4f79-b727-9911a2b363fc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:55.355135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:55.355727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:55.356003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:55.356269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:55.357865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:55.453832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:55.455381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:55.458852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:55.473718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 742/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:55 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:07:55.518863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35973bf6-62be-4d4b-a7f2-b2fc8b76d6ed None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:55.519152 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35973bf6-62be-4d4b-a7f2-b2fc8b76d6ed None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:55.519429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35973bf6-62be-4d4b-a7f2-b2fc8b76d6ed None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:55.520685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 764/1506] 10.222.0.27 () {52 vars in 830 bytes} [Thu Sep 29 02:07:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:55.524232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:55.525452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:55.621851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24f8c9-f2b2-4034-ac40-d2eb2646db01 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:55.713147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:55.713700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:55.713978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:55.714246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] PATH_INFO: `/v3/users/0226aa5f8a8e4cc29a5c70cec8b2366a` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:55.726797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0226aa5f8a8e4cc29a5c70cec8b2366a)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:55.727812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:55.727812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:07:55.727812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:07:55.729084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:55.738487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:55.793424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0226aa5f8a8e4cc29a5c70cec8b2366a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:55.808177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0226aa5f8a8e4cc29a5c70cec8b2366a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:55.808744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0226aa5f8a8e4cc29a5c70cec8b2366a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:55.809154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0226aa5f8a8e4cc29a5c70cec8b2366a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:55.818307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0226aa5f8a8e4cc29a5c70cec8b2366a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:55.846601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 743/1507] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:07:55 2022] DELETE /identity/v3/users/0226aa5f8a8e4cc29a5c70cec8b2366a => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:07:55.851043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:55.851537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:55.851659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:55.908813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:55.910037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:55.983545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:55.996002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 765/1508] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:07:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:56.005691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:56.006916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:56.105772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b19989-9f21-43cc-8fa0-b5e9735fbf43 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:56.158181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:56.177461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:56.178061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:56.178339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:56.178603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:56.180191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:56.284612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:56.286081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3442c25d-06b3-4919-b66a-6c86f9ecaed9 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:56.304416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 744/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:56.644126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:56.645394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:56.657577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235e723d-1056-4745-baf6-a072365bcb45 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:56.755917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:56.756549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:56.756826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:56.757126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] PATH_INFO: `/v3/projects/6649e2e20a344606b65f5ebc4d5479fa` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:56.768194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6649e2e20a344606b65f5ebc4d5479fa)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:56.769264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:56.769264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:07:56.769264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:07:56.770551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:56.829335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6649e2e20a344606b65f5ebc4d5479fa', 'request_id': 'req-28e7a1a1-64cb-45d8-8950-323204a0cb52'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:56.838813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6649e2e20a344606b65f5ebc4d5479fa', 'request_id': 'req-28e7a1a1-64cb-45d8-8950-323204a0cb52'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:56.844857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6649e2e20a344606b65f5ebc4d5479fa', 'request_id': 'req-28e7a1a1-64cb-45d8-8950-323204a0cb52'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:56.873666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] The token cache is being invalidate because project 6649e2e20a344606b65f5ebc4d5479fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:07:56.874258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:56.874362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-28e7a1a1-64cb-45d8-8950-323204a0cb52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:07:56.875713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 766/1510] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:07:56 2022] DELETE /identity/v3/projects/6649e2e20a344606b65f5ebc4d5479fa => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:08:06.302214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=62 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:08:06.871675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:20.310843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b8255-e055-425b-9584-68805256a93e None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:20.311474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b8255-e055-425b-9584-68805256a93e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:20.311474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b8255-e055-425b-9584-68805256a93e None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:20.312816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 745/1511] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:09:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:20.319703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea95c7d3-2eff-49de-9aef-83168f9299b9 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:20.320164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea95c7d3-2eff-49de-9aef-83168f9299b9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:20.320268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea95c7d3-2eff-49de-9aef-83168f9299b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:20.402377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ea95c7d3-2eff-49de-9aef-83168f9299b9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:20.403637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ea95c7d3-2eff-49de-9aef-83168f9299b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:20.504484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 767/1512] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:09:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:09:20.659301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b8255-e055-425b-9584-68805256a93e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:20.661917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f46b8255-e055-425b-9584-68805256a93e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:20.777664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b8255-e055-425b-9584-68805256a93e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:20.783132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c616af-f942-4adc-8c11-f2f8f8ed2420 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:20.783735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c616af-f942-4adc-8c11-f2f8f8ed2420 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:20.784014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c616af-f942-4adc-8c11-f2f8f8ed2420 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:20.784290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c616af-f942-4adc-8c11-f2f8f8ed2420 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:20.792851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c616af-f942-4adc-8c11-f2f8f8ed2420 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:20.794450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c616af-f942-4adc-8c11-f2f8f8ed2420 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:20.804275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 746/1513] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:21.887174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a7f591-e464-43ad-96bb-05fc28455db4 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:21.887689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a7f591-e464-43ad-96bb-05fc28455db4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:21.887689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a7f591-e464-43ad-96bb-05fc28455db4 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:21.889029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 768/1514] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:09:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:21.895186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1088e-1a43-4413-a043-d8f4b75f1f50 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:21.895606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1088e-1a43-4413-a043-d8f4b75f1f50 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:21.895734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1088e-1a43-4413-a043-d8f4b75f1f50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:21.915368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-eed1088e-1a43-4413-a043-d8f4b75f1f50 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:21.916565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eed1088e-1a43-4413-a043-d8f4b75f1f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:21.924303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 747/1515] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:09:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:22.076779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a7f591-e464-43ad-96bb-05fc28455db4 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:22.091122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a7f591-e464-43ad-96bb-05fc28455db4 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:22.096414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f1396a-6453-4b32-9359-187d921b5516 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:22.097026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f1396a-6453-4b32-9359-187d921b5516 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:22.097304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f1396a-6453-4b32-9359-187d921b5516 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:22.097572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f1396a-6453-4b32-9359-187d921b5516 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:22.105892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f1396a-6453-4b32-9359-187d921b5516 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:22.107454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f1396a-6453-4b32-9359-187d921b5516 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:22.117188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 769/1516] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:22.376720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6714bb-1ba1-4c47-8610-20bdc5e4d9f3 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:22.377601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6714bb-1ba1-4c47-8610-20bdc5e4d9f3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:22.377601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6714bb-1ba1-4c47-8610-20bdc5e4d9f3 None None] PATH_INFO: `/v3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:22.378685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 748/1517] 10.222.0.57 () {54 vars in 925 bytes} [Thu Sep 29 02:09:22 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:09:22.385328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dabb6-847a-4e0d-8d49-7991dbd9b622 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:22.385622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dabb6-847a-4e0d-8d49-7991dbd9b622 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:22.385904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dabb6-847a-4e0d-8d49-7991dbd9b622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:22.444959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0b9dabb6-847a-4e0d-8d49-7991dbd9b622 None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:22.446137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0b9dabb6-847a-4e0d-8d49-7991dbd9b622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:22.462228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:22.516389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 770/1518] 10.222.0.57 () {58 vars in 1015 bytes} [Thu Sep 29 02:09:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:26.020938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d42965-a38a-401d-8582-f6b157d550fc None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:26.021356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d42965-a38a-401d-8582-f6b157d550fc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:26.021479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d42965-a38a-401d-8582-f6b157d550fc None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:26.022777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 749/1519] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 29 02:09:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:26.027563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf07cd22-053f-4ea3-af45-7b4da9597dbd None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:26.027995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf07cd22-053f-4ea3-af45-7b4da9597dbd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:26.028099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf07cd22-053f-4ea3-af45-7b4da9597dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:26.047498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cf07cd22-053f-4ea3-af45-7b4da9597dbd None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:26.048673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cf07cd22-053f-4ea3-af45-7b4da9597dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:26.057984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 771/1520] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 29 02:09:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:26.065195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d42965-a38a-401d-8582-f6b157d550fc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:26.078345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d42965-a38a-401d-8582-f6b157d550fc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:26.083656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b56b170-26c8-4ecc-8689-e943cc2125e6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:26.084232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56b170-26c8-4ecc-8689-e943cc2125e6 service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:26.084507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56b170-26c8-4ecc-8689-e943cc2125e6 service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:26.084772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56b170-26c8-4ecc-8689-e943cc2125e6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:26.092671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56b170-26c8-4ecc-8689-e943cc2125e6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:26.094252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56b170-26c8-4ecc-8689-e943cc2125e6 service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:26.104357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 750/1521] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:33.560630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac2317-0ab6-4e02-8f2b-5228ef83a04b None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:33.561314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac2317-0ab6-4e02-8f2b-5228ef83a04b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:33.561314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac2317-0ab6-4e02-8f2b-5228ef83a04b None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:33.562612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 772/1522] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:09:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:33.569228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36bcf08-4e12-45f9-b364-c0fa3d53fdcc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:33.569719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36bcf08-4e12-45f9-b364-c0fa3d53fdcc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:33.569827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36bcf08-4e12-45f9-b364-c0fa3d53fdcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:33.590455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b36bcf08-4e12-45f9-b364-c0fa3d53fdcc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:33.591722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b36bcf08-4e12-45f9-b364-c0fa3d53fdcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:33.600378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 751/1523] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:09:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:33.762832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac2317-0ab6-4e02-8f2b-5228ef83a04b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:33.778069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac2317-0ab6-4e02-8f2b-5228ef83a04b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:33.783461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdccfab8-6e15-4583-8064-e0fb47c15e08 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:33.784050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdccfab8-6e15-4583-8064-e0fb47c15e08 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:33.784328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdccfab8-6e15-4583-8064-e0fb47c15e08 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:33.784604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdccfab8-6e15-4583-8064-e0fb47c15e08 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:33.793082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdccfab8-6e15-4583-8064-e0fb47c15e08 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:33.794654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdccfab8-6e15-4583-8064-e0fb47c15e08 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:33.803779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 773/1524] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:34.871528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbb830-aaf0-4f8f-949f-033bea54d476 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:34.872129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbb830-aaf0-4f8f-949f-033bea54d476 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:34.872129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbb830-aaf0-4f8f-949f-033bea54d476 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:34.873438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 752/1525] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:09:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:34.879947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef41a3-a7d2-402b-a15d-bb25641bf868 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:34.880421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef41a3-a7d2-402b-a15d-bb25641bf868 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:34.880527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef41a3-a7d2-402b-a15d-bb25641bf868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:34.899555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-57ef41a3-a7d2-402b-a15d-bb25641bf868 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:34.900759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-57ef41a3-a7d2-402b-a15d-bb25641bf868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:34.908607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 774/1526] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:09:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:35.061532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecbb830-aaf0-4f8f-949f-033bea54d476 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:35.075831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecbb830-aaf0-4f8f-949f-033bea54d476 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:35.081231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d087b3b-0b07-414e-8597-089ce77192d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:35.081815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d087b3b-0b07-414e-8597-089ce77192d1 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:35.082092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d087b3b-0b07-414e-8597-089ce77192d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:35.082360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d087b3b-0b07-414e-8597-089ce77192d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:35.090763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d087b3b-0b07-414e-8597-089ce77192d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:35.092381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d087b3b-0b07-414e-8597-089ce77192d1 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:35.102457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 753/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:38.945010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a86b92-b7a9-45a9-97b0-1856bb8eb9a4 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:38.945533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a86b92-b7a9-45a9-97b0-1856bb8eb9a4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:38.945533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a86b92-b7a9-45a9-97b0-1856bb8eb9a4 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:38.946874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 775/1528] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:09:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:38.952133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2478f2e9-9b5d-4a54-ac84-a1ab117173c6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:38.952616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2478f2e9-9b5d-4a54-ac84-a1ab117173c6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:38.952723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2478f2e9-9b5d-4a54-ac84-a1ab117173c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:38.972186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2478f2e9-9b5d-4a54-ac84-a1ab117173c6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:38.973403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2478f2e9-9b5d-4a54-ac84-a1ab117173c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:38.981690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 754/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:09:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:39.269211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a86b92-b7a9-45a9-97b0-1856bb8eb9a4 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:39.270920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-33a86b92-b7a9-45a9-97b0-1856bb8eb9a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:39.346942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a86b92-b7a9-45a9-97b0-1856bb8eb9a4 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:39.350505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c101ffe-78c8-4768-906e-1508788ed15f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:39.351106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c101ffe-78c8-4768-906e-1508788ed15f service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:39.351381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c101ffe-78c8-4768-906e-1508788ed15f service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:39.351655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c101ffe-78c8-4768-906e-1508788ed15f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:39.360706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c101ffe-78c8-4768-906e-1508788ed15f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:39.362156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c101ffe-78c8-4768-906e-1508788ed15f service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:39.371587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 776/1530] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:41.511744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed79684-c13a-4b16-abbc-391d40904405 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:41.512332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed79684-c13a-4b16-abbc-391d40904405 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:41.512332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed79684-c13a-4b16-abbc-391d40904405 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:41.513633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 755/1531] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:09:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:41.518877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d2348-4c5c-4cf8-a0a5-91c94ca83a0e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:41.519322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d2348-4c5c-4cf8-a0a5-91c94ca83a0e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:41.519429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316d2348-4c5c-4cf8-a0a5-91c94ca83a0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:41.539293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-316d2348-4c5c-4cf8-a0a5-91c94ca83a0e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:41.540472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-316d2348-4c5c-4cf8-a0a5-91c94ca83a0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:41.548103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 777/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:09:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:41.839415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed79684-c13a-4b16-abbc-391d40904405 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:41.841129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fed79684-c13a-4b16-abbc-391d40904405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:41.854513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed79684-c13a-4b16-abbc-391d40904405 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:41.858149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4e9ca8-11e0-433d-953b-f64ad8b5ac41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:41.858738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4e9ca8-11e0-433d-953b-f64ad8b5ac41 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:41.859019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4e9ca8-11e0-433d-953b-f64ad8b5ac41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:41.859291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4e9ca8-11e0-433d-953b-f64ad8b5ac41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:41.868991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4e9ca8-11e0-433d-953b-f64ad8b5ac41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:41.870409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4e9ca8-11e0-433d-953b-f64ad8b5ac41 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:41.879578 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 756/1533] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:09:45.474362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd09ee6-9e5b-4c40-8eb2-466e4539c842 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:45.474947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd09ee6-9e5b-4c40-8eb2-466e4539c842 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:45.474947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd09ee6-9e5b-4c40-8eb2-466e4539c842 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:45.476258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 778/1534] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:09:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:45.481385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba25967-72cf-4565-a367-0c984360f650 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:45.481821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba25967-72cf-4565-a367-0c984360f650 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:45.481929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba25967-72cf-4565-a367-0c984360f650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:45.502292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-dba25967-72cf-4565-a367-0c984360f650 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:45.503478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dba25967-72cf-4565-a367-0c984360f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:45.511135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 757/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:09:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:45.814679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd09ee6-9e5b-4c40-8eb2-466e4539c842 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:45.816669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1bd09ee6-9e5b-4c40-8eb2-466e4539c842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:45.880766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:45.894220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd09ee6-9e5b-4c40-8eb2-466e4539c842 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:45.898269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23b477d-cd12-4a14-902a-e70f1a19404f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:45.898861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b477d-cd12-4a14-902a-e70f1a19404f service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:45.899134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b477d-cd12-4a14-902a-e70f1a19404f service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:45.899404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b477d-cd12-4a14-902a-e70f1a19404f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:45.908010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23b477d-cd12-4a14-902a-e70f1a19404f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:45.909463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23b477d-cd12-4a14-902a-e70f1a19404f service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:45.919169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 779/1536] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:48.055242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f303a7e-4578-4239-b87a-15cb9df511fa None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:48.055824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f303a7e-4578-4239-b87a-15cb9df511fa None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:48.055824 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f303a7e-4578-4239-b87a-15cb9df511fa None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:48.057213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 758/1537] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:09:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:48.062823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f892d67-db8c-4a43-92f8-ee9178b03293 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:48.063291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f892d67-db8c-4a43-92f8-ee9178b03293 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:48.063399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f892d67-db8c-4a43-92f8-ee9178b03293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:48.083716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9f892d67-db8c-4a43-92f8-ee9178b03293 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:48.084935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9f892d67-db8c-4a43-92f8-ee9178b03293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:48.092768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 780/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:48.408380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f303a7e-4578-4239-b87a-15cb9df511fa None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:48.410123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1f303a7e-4578-4239-b87a-15cb9df511fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:48.418222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=84 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:48.428507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f303a7e-4578-4239-b87a-15cb9df511fa None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:48.432015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1794421-fb7a-43d8-90ce-b5cf2559ab2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:48.432626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1794421-fb7a-43d8-90ce-b5cf2559ab2a service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:48.432915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1794421-fb7a-43d8-90ce-b5cf2559ab2a service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:48.433188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1794421-fb7a-43d8-90ce-b5cf2559ab2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:48.443028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1794421-fb7a-43d8-90ce-b5cf2559ab2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:48.444444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1794421-fb7a-43d8-90ce-b5cf2559ab2a service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:48.453842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 759/1539] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:58.078717 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=11 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:58.449608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:12.583480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.584566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.585089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.612384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:12.614079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:12.626859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 781/1540] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:12.637199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.637782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.638111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.670639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:12.671842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:12.687603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 760/1541] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:12.693062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:12.707589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf639f1-0aa7-47bd-a632-a17a5ff0cbb6 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:12.713941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:12.715529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.716041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.716557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.717519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:12.718083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:12.720047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:12.739100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 782/1542] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:12.749316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:12.764906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439c6d98-b43a-4f66-b0fd-44d4d50c56d3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:12.773775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960eee00-1944-42e6-b797-36cba854856c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:12.774426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960eee00-1944-42e6-b797-36cba854856c admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.774705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960eee00-1944-42e6-b797-36cba854856c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.774978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960eee00-1944-42e6-b797-36cba854856c admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.775660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960eee00-1944-42e6-b797-36cba854856c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:12.780790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960eee00-1944-42e6-b797-36cba854856c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:12.845839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 761/1543] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:12 2022] POST /identity/v3/projects => generated 377 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:12.851825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:12.867379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648029a-5962-412d-a0e4-98eb95d480c9 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:12.912429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76b722-55df-4ade-92de-a542e326e23e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.912707 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76b722-55df-4ade-92de-a542e326e23e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.913273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76b722-55df-4ade-92de-a542e326e23e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.945472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2c76b722-55df-4ade-92de-a542e326e23e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:12.947242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2c76b722-55df-4ade-92de-a542e326e23e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:12.947744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:12.948378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.948661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.948962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.949601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:12.956892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:12.970754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 762/1544] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:12.980028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:12.980674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:12.981230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:12.989186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 783/1545] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:12 2022] POST /identity/v3/users => generated 289 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:12.993352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.008551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c3e9aa-a8ed-4ffc-9586-e9d5cd3c79fc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.013938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.014273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.014890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.015158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.015280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.015452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.016428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.018052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.035873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 763/1546] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.039051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 784/1547] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:13.040299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.042784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.063522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c5a0ad-612e-41c7-a0f7-ed66e5c7abd4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.068687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed48f769-0c61-43bc-96d0-1609cb6ba329 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.075181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.075912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.076193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.076463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.077138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.077439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.078562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:11:13.078562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:11:13.078562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:11:13.079303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef980c8c-1823-44d8-9d6a-2b59beba2228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.079438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.079917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef980c8c-1823-44d8-9d6a-2b59beba2228 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.080198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef980c8c-1823-44d8-9d6a-2b59beba2228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.080476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef980c8c-1823-44d8-9d6a-2b59beba2228 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.081385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef980c8c-1823-44d8-9d6a-2b59beba2228 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.082979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef980c8c-1823-44d8-9d6a-2b59beba2228 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.102996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 785/1548] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:11:13.103328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 764/1549] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:11:13.104286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.104575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.104853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.111205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.132057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35017ef-334c-4202-b7fe-2b642e101cd4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.139308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.140545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.142679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9cdf2a-6384-454f-836f-438c7620155c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.143306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9cdf2a-6384-454f-836f-438c7620155c admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.143587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9cdf2a-6384-454f-836f-438c7620155c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.143861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9cdf2a-6384-454f-836f-438c7620155c admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.144495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9cdf2a-6384-454f-836f-438c7620155c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.146726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9cdf2a-6384-454f-836f-438c7620155c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.164165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 786/1550] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.165040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.182767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3a2f9b-da51-4a55-9802-b3333a6c3204 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.193457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.194159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.194473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.194758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] PATH_INFO: `/v3/projects/dbda18c4ee4e4c9eb64f37e71bd862cd/users/2ab3ddf040e148789b3c67fc23d2da3d/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.201940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 765/1551] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/projects => generated 375 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:11:13.203425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a900ec5-c95e-4fbb-abb1-deb78a9681cc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.203785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a900ec5-c95e-4fbb-abb1-deb78a9681cc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.205741 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a900ec5-c95e-4fbb-abb1-deb78a9681cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.217854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbda18c4ee4e4c9eb64f37e71bd862cd, user_id=2ab3ddf040e148789b3c67fc23d2da3d, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.219287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:11:13.219287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:11:13.219287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:11:13.221952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470aa7b6-7262-46a4-994f-f0cfe13daa34 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.236765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0a900ec5-c95e-4fbb-abb1-deb78a9681cc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.238028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0a900ec5-c95e-4fbb-abb1-deb78a9681cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.240568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 787/1552] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:13 2022] PUT /identity/v3/projects/dbda18c4ee4e4c9eb64f37e71bd862cd/users/2ab3ddf040e148789b3c67fc23d2da3d/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:13.241922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dfe548-4bb0-40b9-b468-3bb9ebaa14a6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.242326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dfe548-4bb0-40b9-b468-3bb9ebaa14a6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.242605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dfe548-4bb0-40b9-b468-3bb9ebaa14a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.263142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-35dfe548-4bb0-40b9-b468-3bb9ebaa14a6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.264354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-35dfe548-4bb0-40b9-b468-3bb9ebaa14a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.300392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 766/1553] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.301758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22770b36-f1d2-45a9-8050-bdd9f87092d1 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.302632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22770b36-f1d2-45a9-8050-bdd9f87092d1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.302912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22770b36-f1d2-45a9-8050-bdd9f87092d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.322097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-22770b36-f1d2-45a9-8050-bdd9f87092d1 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.323295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-22770b36-f1d2-45a9-8050-bdd9f87092d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.324844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 788/1554] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.326091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.326370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.326668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.334935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 767/1555] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.336276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b992eb09-9d5b-4f34-8a5b-e2dd013d542e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.336563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b992eb09-9d5b-4f34-8a5b-e2dd013d542e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.336835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b992eb09-9d5b-4f34-8a5b-e2dd013d542e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.346743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.347951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.356543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b992eb09-9d5b-4f34-8a5b-e2dd013d542e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.356698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 789/1556] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.357428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.357832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b992eb09-9d5b-4f34-8a5b-e2dd013d542e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.366350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 768/1557] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.367564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9917ad95-2c25-4600-afe2-f72867ee74ff None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.367683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783697a6-af1f-4806-bbe8-a119a12a4776 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.367879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783697a6-af1f-4806-bbe8-a119a12a4776 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.368223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783697a6-af1f-4806-bbe8-a119a12a4776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.376038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7406e174-22b5-48f8-8722-08008c2a1420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.376635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7406e174-22b5-48f8-8722-08008c2a1420 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.376937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7406e174-22b5-48f8-8722-08008c2a1420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.377214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7406e174-22b5-48f8-8722-08008c2a1420 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.378255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7406e174-22b5-48f8-8722-08008c2a1420 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.379780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7406e174-22b5-48f8-8722-08008c2a1420 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.388189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-783697a6-af1f-4806-bbe8-a119a12a4776 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.389587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-783697a6-af1f-4806-bbe8-a119a12a4776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.390734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:13.399750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 769/1558] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.401046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.401294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.401593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.407447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 790/1559] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/users => generated 288 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:13.407447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e230ec6-32c6-4b39-a02e-a31186872912 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.407447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e230ec6-32c6-4b39-a02e-a31186872912 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.407447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e230ec6-32c6-4b39-a02e-a31186872912 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.420556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.421780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.426144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1e230ec6-32c6-4b39-a02e-a31186872912 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.427839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1e230ec6-32c6-4b39-a02e-a31186872912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.429967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 770/1560] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.430729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.440722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b2dc3-d743-4e47-9ec9-2454901eda5e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.440863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 791/1561] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.441976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c365224b-99b2-4558-a964-3e7ba94d70f7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.442271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c365224b-99b2-4558-a964-3e7ba94d70f7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.442592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c365224b-99b2-4558-a964-3e7ba94d70f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.447070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.447667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.448071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.448353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.449030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.449337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.450935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df2b3cd-f2ef-4c54-9cbb-95b8f9bd72fd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.462337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c365224b-99b2-4558-a964-3e7ba94d70f7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.463251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 771/1562] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:13.463500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c365224b-99b2-4558-a964-3e7ba94d70f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.464515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1f26db-0ef7-4174-9b13-adb62049dbaf None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.464802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1f26db-0ef7-4174-9b13-adb62049dbaf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.465106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1f26db-0ef7-4174-9b13-adb62049dbaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.476738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 792/1563] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.480490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.480490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.480490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.486986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3e1f26db-0ef7-4174-9b13-adb62049dbaf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.488182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3e1f26db-0ef7-4174-9b13-adb62049dbaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.500725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 772/1564] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.501283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.501418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.501418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.501418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.501643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.507910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 793/1565] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.508546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.518075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce1714f-49eb-4563-acbc-7d295f84e9c1 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.522642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.522793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:13.523229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.523506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.523779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.523915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:13.524449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.526030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.531869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 773/1566] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:13.532767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.539564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 794/1567] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:13.539564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.547253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba43783-c711-47df-bd8e-8383bc09cd8d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.551778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.552931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.553242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.553514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.554203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.554506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.556080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.561404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:13.577314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 795/1568] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:13.577314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.591656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04b3b43-baeb-4d4d-a84c-81b4eb919a7b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.597486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.598086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.598362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.598630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.599257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.600263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:11:13.600263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:11:13.600263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:11:13.601490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.602731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30c711-bf40-47ce-a56d-baf3e940eae5 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.606340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.607042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.607210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.607478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.617201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.618653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.628232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 774/1569] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:13.629173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 796/1570] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:13.629308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.633613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.639120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aff6f59-00be-4994-a836-3c867ea35ddb service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.706806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.707537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.707671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab671ca7-565a-477d-a57d-07c2de452c63 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.707830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.708586 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.708784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.709175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.710765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.713260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.713982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.714436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.714436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.715102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.715459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.717164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.725488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 775/1571] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:13.726414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.736712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f19fc8-c58f-4557-98e6-14b6523dc0a2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.741034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 797/1572] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:13.742338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.742705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.743379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.743646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.743917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.744556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.745017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.746610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.756559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4fd978-64a5-4040-9511-7028fcba2874 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.760643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 776/1573] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:13.761643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.769863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.770823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.771309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.772519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.773413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:13.773891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.774449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d29c0f3-56ab-47a0-b141-6f4393a5528f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.776139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.779929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.780526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.780793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.781099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.781738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.783319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.795056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 777/1574] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:13.799057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.799380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 798/1575] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:13.801408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.814546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9460d51e-cf52-493d-8bd1-4c49b0d5835d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.821451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b18cb1-4027-4340-8d42-45c991851092 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.821751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.822464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.822749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.823206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.823811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.826124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.827579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.828323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.828687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.829060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.829749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.832373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.860729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 799/1576] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/users => generated 293 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:13.860729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.868139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 778/1577] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/projects => generated 391 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:13.869341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:13.869957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36987bfe-7f18-4e5d-b618-676a0f3d3dcc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.932585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:13.947143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.947725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5574ba9d-73b3-4982-95d0-45c651d0e539 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:13.948287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.948800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.949376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.950627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.953639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.956616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:13.957412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:13.957696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:13.957974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:13.958724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:13.960685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:13.990870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:13.994382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 800/1578] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/projects => generated 373 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:13.996250 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.002739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 779/1579] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/projects => generated 361 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.004532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.067697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4054459b-6076-4d5d-8763-15a6f00f90e9 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.082649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a779ee5-38f3-466f-be34-2cf732a23ef5 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.088945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.089944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.090275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.090536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] PATH_INFO: `/v3/projects/2a04f2488c2a4d219ad8193c4dd12b65/users/066c7318255e4a01a0f6a81715718e15/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.107916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a04f2488c2a4d219ad8193c4dd12b65, user_id=066c7318255e4a01a0f6a81715718e15, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.111070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.126820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 780/1580] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:14 2022] PUT /identity/v3/projects/2a04f2488c2a4d219ad8193c4dd12b65/users/066c7318255e4a01a0f6a81715718e15/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:14.127922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.146347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.146961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.147234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.147506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.148127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.150052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.181467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 801/1581] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:13 2022] POST /identity/v3/projects => generated 375 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.182339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.201797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954b4aa-d1ca-4c4c-8a42-88e501d3922a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.255081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fedfe8b-4136-411c-9412-3fae951d9d40 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.260334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.260970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.261249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.261519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.262165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.264382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.275248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 802/1582] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.276237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.278610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.279895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.279895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.280397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.281105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.283526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.283786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:14.289479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab339a-d61b-4b10-b5ff-7038dda6ebdd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.306397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.307173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.308794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.309269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.310046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.312921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.321196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 781/1583] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/projects => generated 383 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.322019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.337754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:14.342689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 803/1584] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/users => generated 296 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.344100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.359215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb6065c-2770-4de8-a6ef-66a5fe55e070 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.402540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98395bcb-c1ad-44b8-998c-f42f361ceb4d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.412259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.413724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.414181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.414671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.415519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.417284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:11:14.417284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:11:14.417284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:11:14.419357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a783cc77-3c5a-4ed2-aeab-d8810e196536 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.436760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.437459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.437878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.438278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.438934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.441865 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.452813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 782/1585] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/users => generated 287 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.454312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.454766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.455183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.478750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 804/1586] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/users => generated 281 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.483481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.498919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:14.498919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:14.504122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f1175d-7974-4e8b-84bc-dfcf355fd37b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.509754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.510283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 783/1587] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:14.511314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.521549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.521880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.522121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.522883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.524196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423182b6-83d8-44d7-bdaa-1cccf5b66a19 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.525185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.529640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.530668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.531211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.531681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.532751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.535081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.552461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 805/1588] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/users => generated 288 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.553263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.555555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 784/1589] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.559646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.572010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4807906e-081c-43a4-a761-b71b3e118c34 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.577941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c886ca-d452-4ee2-9fec-fc4af6c18f44 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.585085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.586008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.586506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.587014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.587663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.588587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.588668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.588911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.589616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.590139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.591988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.592245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.609876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 785/1590] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.611593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.629414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 806/1591] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/users => generated 292 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:14.633290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1fab18-8704-4f6b-ad26-9c088fa70e69 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.633863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.639772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.641116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.641465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.641979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.642747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.645251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.647013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4dd6ca-f0f9-44db-8eb5-9c4265c7739d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.652556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.653439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.653925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.654405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.655242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.657081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.659776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 786/1592] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.664178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.668898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 807/1593] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.670162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.682461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b958f384-620c-484f-aa06-82ad01fe044e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.688845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.689848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.690985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.691449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.692276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.694748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.713445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 808/1594] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.713445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.724724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f5110c-42a1-4a4e-b3ec-0762b47fe954 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.730282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.731098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.732091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.732476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] PATH_INFO: `/v3/projects/6329c07d43e4411989cd06fae61982e4/users/acfdbe69a1be4bac946a5c99cc9bbc56/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.735951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5629d4-0464-4c7e-8293-9c105c420dc6 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.740599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.741254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.741708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.741982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.752332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.753912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.755138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6329c07d43e4411989cd06fae61982e4, user_id=acfdbe69a1be4bac946a5c99cc9bbc56, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.758156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.765507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 787/1595] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:14.766379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.777458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9e1830-e159-4455-915b-6c635607e81d service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.777764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 809/1596] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:14 2022] PUT /identity/v3/projects/6329c07d43e4411989cd06fae61982e4/users/acfdbe69a1be4bac946a5c99cc9bbc56/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:14.778968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.871293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.872495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.872779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.873096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.875215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.877201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.890532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=87,keystone:INSERT=13 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:14.901282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a13a3e1-38d6-4cc5-96e4-d13d2ba82165 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.901500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 788/1597] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.901500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.908145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.908934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.909317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.909848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.910806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.913656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.916985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:14.926586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 810/1598] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.927752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.928248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeedafe2-24f1-489d-b740-b39c2034dd52 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.947459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f548e1-907c-41ce-826a-8ad04fc73ae5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.947906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0399d9-c5a7-440c-b6c4-a89979856621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.949148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0399d9-c5a7-440c-b6c4-a89979856621 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.949675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0399d9-c5a7-440c-b6c4-a89979856621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.950574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0399d9-c5a7-440c-b6c4-a89979856621 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.951442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0399d9-c5a7-440c-b6c4-a89979856621 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.953373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.954204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.954323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.955426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.956089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.958682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.959039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0399d9-c5a7-440c-b6c4-a89979856621 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:14.972562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 811/1599] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.973669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:14.985871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6d6ff9-6c90-4a7b-8ffd-65078351cd66 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:14.986464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 789/1600] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:14.988204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.988722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.989190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.992506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:14.993291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:14.993587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:14.993866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:14.994522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:14.996121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.009092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 812/1601] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:14 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:15.013426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.027285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efffadf-b5f6-4d94-842c-e965de0ef089 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.032487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] MFA Rules not processed for user `acfdbe69a1be4bac946a5c99cc9bbc56`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:15.033751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.034435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.034747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.035170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] PATH_INFO: `/v3/projects/320dd138f254473a84a601c153e1e39b/users/53c020af94d64bac81df7837a86586b2/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.038566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:15.055477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=320dd138f254473a84a601c153e1e39b, user_id=53c020af94d64bac81df7837a86586b2, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:15.058485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.082530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 813/1602] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:15 2022] PUT /identity/v3/projects/320dd138f254473a84a601c153e1e39b/users/53c020af94d64bac81df7837a86586b2/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:15.086197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.161959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 790/1603] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:15.162915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.186636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b82e21-7d19-4a51-ac29-0946cfb4908d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.210238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c291ae16-7c87-46c5-8488-d2bceda4fcc7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.240107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.241562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.246082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.246733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.247768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:15.250217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.287240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 814/1604] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:15 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:15.289056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.329349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479fcc8-ba00-497b-bf24-edb2a75f7704 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.345088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.345653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.346012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.346384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] PATH_INFO: `/v3/projects/77947ca6b22e4f03b17317021a2bb7fa/users/2097a3ea6f7249bbbd496d49da9beb31/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.365691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f33e5-f1c3-42ab-a3ca-1ef9f26ed981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.366520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f33e5-f1c3-42ab-a3ca-1ef9f26ed981 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.367241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f33e5-f1c3-42ab-a3ca-1ef9f26ed981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.367395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f33e5-f1c3-42ab-a3ca-1ef9f26ed981 admin admin] PATH_INFO: `/v3/projects/62c01a340fd24c00ba44152b74b8ef76/users/cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.386327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] RBAC: Authorizing `identity:create_grant(project_id=77947ca6b22e4f03b17317021a2bb7fa, user_id=2097a3ea6f7249bbbd496d49da9beb31, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:15.389190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.397598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f33e5-f1c3-42ab-a3ca-1ef9f26ed981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62c01a340fd24c00ba44152b74b8ef76, user_id=cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:15.401243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f33e5-f1c3-42ab-a3ca-1ef9f26ed981 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.429046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 815/1605] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:15 2022] PUT /identity/v3/projects/77947ca6b22e4f03b17317021a2bb7fa/users/2097a3ea6f7249bbbd496d49da9beb31/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:15.429046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.429373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 791/1606] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:15 2022] PUT /identity/v3/projects/62c01a340fd24c00ba44152b74b8ef76/users/cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:15.429373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab98a83a-f2d6-4269-8bdf-e4f5db5fa96d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.429373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab98a83a-f2d6-4269-8bdf-e4f5db5fa96d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.429373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab98a83a-f2d6-4269-8bdf-e4f5db5fa96d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.454344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ab98a83a-f2d6-4269-8bdf-e4f5db5fa96d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:15.456396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ab98a83a-f2d6-4269-8bdf-e4f5db5fa96d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:15.568224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a5c004-9598-48ac-89d5-4b82945beeae admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.591673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 792/1607] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:15.593744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf896ff-e652-4b3f-aed1-426a3ff51cee None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.594117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf896ff-e652-4b3f-aed1-426a3ff51cee None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.594542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf896ff-e652-4b3f-aed1-426a3ff51cee None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.601290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 793/1608] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 29 02:11:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:15.602475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf896ff-e652-4b3f-aed1-426a3ff51cee None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.610433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:15.620040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:15.629216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf896ff-e652-4b3f-aed1-426a3ff51cee None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.636799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f465e9-f59c-45b9-aadc-9b7defd9d5be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.637630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f465e9-f59c-45b9-aadc-9b7defd9d5be admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.637909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f465e9-f59c-45b9-aadc-9b7defd9d5be admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.638180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f465e9-f59c-45b9-aadc-9b7defd9d5be admin admin] PATH_INFO: `/v3/projects/cc20488e0b2d48f796052c4e76990b19/users/f16c1cde815c4a2580cd5ebf7b3ed76d/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.658019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f465e9-f59c-45b9-aadc-9b7defd9d5be admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc20488e0b2d48f796052c4e76990b19, user_id=f16c1cde815c4a2580cd5ebf7b3ed76d, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:15.661182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f465e9-f59c-45b9-aadc-9b7defd9d5be admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.674987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2ef23c-2c08-4f8e-bef7-71319156c027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.676363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2ef23c-2c08-4f8e-bef7-71319156c027 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.678328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2ef23c-2c08-4f8e-bef7-71319156c027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.678621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2ef23c-2c08-4f8e-bef7-71319156c027 admin admin] PATH_INFO: `/v3/projects/1f133b5228214aa2bbf9b7d5bd35785b/users/0580c757f98549f992eed1e8ff96f896/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.694847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 794/1609] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:15 2022] PUT /identity/v3/projects/cc20488e0b2d48f796052c4e76990b19/users/f16c1cde815c4a2580cd5ebf7b3ed76d/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:15.695185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f8c5cd-46b4-4a83-be60-fb739ed2ed4a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.695279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f8c5cd-46b4-4a83-be60-fb739ed2ed4a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.702591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f8c5cd-46b4-4a83-be60-fb739ed2ed4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.709461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2ef23c-2c08-4f8e-bef7-71319156c027 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f133b5228214aa2bbf9b7d5bd35785b, user_id=0580c757f98549f992eed1e8ff96f896, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:15.713427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2ef23c-2c08-4f8e-bef7-71319156c027 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:15.731993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 816/1610] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:15 2022] PUT /identity/v3/projects/1f133b5228214aa2bbf9b7d5bd35785b/users/0580c757f98549f992eed1e8ff96f896/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:15.734022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.734576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.734926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.750279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c3f8c5cd-46b4-4a83-be60-fb739ed2ed4a None None] MFA Rules not processed for user `2097a3ea6f7249bbbd496d49da9beb31`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:15.751694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c3f8c5cd-46b4-4a83-be60-fb739ed2ed4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:15.780090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] MFA Rules not processed for user `cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:15.781623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:15.884375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 817/1611] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:15.885358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:15.909342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 795/1612] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:15.912855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f9dc9-7b03-40a0-b270-6025167e6013 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.913191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f9dc9-7b03-40a0-b270-6025167e6013 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.913772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123f9dc9-7b03-40a0-b270-6025167e6013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:15.977321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-123f9dc9-7b03-40a0-b270-6025167e6013 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:15.977899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be910f0c-2f3a-4fff-b1b3-8a4751fc75ae None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:15.978553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-123f9dc9-7b03-40a0-b270-6025167e6013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:15.983458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af295af0-763f-4023-9e77-6d836b1c8585 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:15.984141 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af295af0-763f-4023-9e77-6d836b1c8585 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:15.985076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af295af0-763f-4023-9e77-6d836b1c8585 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:15.985819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af295af0-763f-4023-9e77-6d836b1c8585 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.086131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:16.131970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af295af0-763f-4023-9e77-6d836b1c8585 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.133491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af295af0-763f-4023-9e77-6d836b1c8585 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.140248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 796/1613] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 29 02:11:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:16.142361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.143429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.144295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.151273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 818/1614] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:16.152559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.152848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.153167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.178584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:16.180314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:16.191455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 797/1615] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:16.194770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.206847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] MFA Rules not processed for user `0580c757f98549f992eed1e8ff96f896`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:16.208113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:16.220068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c89837-622f-401d-a1c1-b2bc47a5d9c5 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.234864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.235878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.236388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.236923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.237868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.240225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.283750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 798/1616] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/projects => generated 365 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:16.292740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.324935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 819/1617] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:16.326203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.344482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba5123d-fa74-432c-9b2c-ce3b151443ec None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.357528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:16.463918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.464290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.464673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.465383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.490253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9dd5d9-6820-4d66-b164-336383b969a0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.515652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.517997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.523257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.523736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.524838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.531074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.637676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 799/1618] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/projects => generated 371 bytes in 347 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:16.641511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.650788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.654479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.674464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 820/1619] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:16 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:16.676001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.697886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:16.787213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496bab1-1e58-4408-b552-a6c5083bf9e7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.796915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d04483-a95d-4b50-b3bd-9620b2c37871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.798740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a254dab-3e81-4923-85dd-85aecf608dbe service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.799843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d04483-a95d-4b50-b3bd-9620b2c37871 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.801035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d04483-a95d-4b50-b3bd-9620b2c37871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.801765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d04483-a95d-4b50-b3bd-9620b2c37871 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.804220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d04483-a95d-4b50-b3bd-9620b2c37871 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.811588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d04483-a95d-4b50-b3bd-9620b2c37871 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.845373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 800/1620] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/users => generated 289 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:16.847371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f7d9f-e2a2-472c-915e-678d101c1913 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.847961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f7d9f-e2a2-472c-915e-678d101c1913 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.849413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8f7d9f-e2a2-472c-915e-678d101c1913 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.852945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 801/1621] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 29 02:11:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:16.854113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8f7d9f-e2a2-472c-915e-678d101c1913 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.874526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8f7d9f-e2a2-472c-915e-678d101c1913 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.883075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.884604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.885130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.885802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.886653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.889628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.946726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 802/1622] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:16.948001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:16.960095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.960439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.960726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.961037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.963587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe2697-76c7-4055-ac7f-d27be279719a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:16.970005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9610c60-8e36-4706-b348-fd16d518b313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:16.971482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9610c60-8e36-4706-b348-fd16d518b313 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.971995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9610c60-8e36-4706-b348-fd16d518b313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.972515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9610c60-8e36-4706-b348-fd16d518b313 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:16.974763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9610c60-8e36-4706-b348-fd16d518b313 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.976815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9610c60-8e36-4706-b348-fd16d518b313 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.982618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:16.985794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:16.993861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 803/1623] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:16.996022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:16.996903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:16.997408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.015186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 821/1624] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:17.015894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.027582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:17.031382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:17.041409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9b19d2-4238-496d-b5bc-6a07fabd6ed2 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.052808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.053552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.053835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.054107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.056053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.059534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.088906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 822/1625] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:17 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:17.088906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.107530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:17.111495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88928e4-f41c-416a-8aa8-f16c3ac553a7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.125580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 804/1626] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 29 02:11:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:17.126804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.127578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb128e5-074b-4c6c-b3de-940906fda6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.128625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb128e5-074b-4c6c-b3de-940906fda6d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.128761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb128e5-074b-4c6c-b3de-940906fda6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.129019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb128e5-074b-4c6c-b3de-940906fda6d7 admin admin] PATH_INFO: `/v3/projects/9a0db52fee0c4d33aef53d9113aab226/users/37ec649f9c1e4d198e61c583e6a495b9/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.142917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee87d52d-eefd-4cfc-bd30-9e1e92e669bd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.150924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.151630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.151942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.153204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] PATH_INFO: `/v3/projects/fb347b0fdfbe4276b34799d40a000c59/users/0e72daa1f1584c47ad375cf8143643b1/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.168456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb128e5-074b-4c6c-b3de-940906fda6d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a0db52fee0c4d33aef53d9113aab226, user_id=37ec649f9c1e4d198e61c583e6a495b9, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.173087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb347b0fdfbe4276b34799d40a000c59, user_id=0e72daa1f1584c47ad375cf8143643b1, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.176069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb128e5-074b-4c6c-b3de-940906fda6d7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.176196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.198879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 805/1627] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:17 2022] PUT /identity/v3/projects/fb347b0fdfbe4276b34799d40a000c59/users/0e72daa1f1584c47ad375cf8143643b1/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:17.198879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.202413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 823/1628] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:17 2022] PUT /identity/v3/projects/9a0db52fee0c4d33aef53d9113aab226/users/37ec649f9c1e4d198e61c583e6a495b9/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:17.206666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c275b23e-0399-49da-95c4-e597a8ac061b None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.206956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c275b23e-0399-49da-95c4-e597a8ac061b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.209239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c275b23e-0399-49da-95c4-e597a8ac061b None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.215013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 824/1629] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 29 02:11:17 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:17.216177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c275b23e-0399-49da-95c4-e597a8ac061b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.295931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4fd7b5-b20e-474c-926f-ab37da7b322f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.369330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c275b23e-0399-49da-95c4-e597a8ac061b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.384562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.385218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.385508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.386029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.386826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.388741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.413686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 825/1630] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:17 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:17.415054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.439464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.440679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.441251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.441747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.442885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc3d830-5637-4c86-9ba2-3cc834fcf2c4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.454182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3bb524-fdab-48be-9aae-0b0150c7716a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.454884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3bb524-fdab-48be-9aae-0b0150c7716a admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.455165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3bb524-fdab-48be-9aae-0b0150c7716a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.455435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3bb524-fdab-48be-9aae-0b0150c7716a admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.456083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3bb524-fdab-48be-9aae-0b0150c7716a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.458792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3bb524-fdab-48be-9aae-0b0150c7716a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.481205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 826/1631] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:17 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:17.491261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.491604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.491839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.517985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.519607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.546109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:17.547365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:17.554333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 806/1632] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:17 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:17.555196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.574724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 827/1633] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:11:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:17.575720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.585860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1fde82-a599-4599-811f-a3a7897e9f6d service nova] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.597180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f32cd9c-d935-4450-a5f8-5b69e2ff0074 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.609788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607d86b9-2012-49fa-a3a2-3fa34fdbabf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.612196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607d86b9-2012-49fa-a3a2-3fa34fdbabf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.612580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.613147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607d86b9-2012-49fa-a3a2-3fa34fdbabf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.613358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.613594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.613866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] PATH_INFO: `/v3/projects/9a0db52fee0c4d33aef53d9113aab226/users/37ec649f9c1e4d198e61c583e6a495b9/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.615111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607d86b9-2012-49fa-a3a2-3fa34fdbabf5 admin admin] PATH_INFO: `/v3/projects/fb347b0fdfbe4276b34799d40a000c59/users/0e72daa1f1584c47ad375cf8143643b1/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.616839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a0db52fee0c4d33aef53d9113aab226, user_id=37ec649f9c1e4d198e61c583e6a495b9, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.619541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.629580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607d86b9-2012-49fa-a3a2-3fa34fdbabf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb347b0fdfbe4276b34799d40a000c59, user_id=0e72daa1f1584c47ad375cf8143643b1, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:17.633839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607d86b9-2012-49fa-a3a2-3fa34fdbabf5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:17.639770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:17.659957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 828/1634] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:17 2022] PUT /identity/v3/projects/9a0db52fee0c4d33aef53d9113aab226/users/37ec649f9c1e4d198e61c583e6a495b9/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:17.659957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:17.668294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:17.671626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 807/1635] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:17 2022] PUT /identity/v3/projects/fb347b0fdfbe4276b34799d40a000c59/users/0e72daa1f1584c47ad375cf8143643b1/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:17.678004 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef92d0a0-e46e-4811-a337-0835261e2eb5 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.678778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef92d0a0-e46e-4811-a337-0835261e2eb5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.679302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef92d0a0-e46e-4811-a337-0835261e2eb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.733632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ef92d0a0-e46e-4811-a337-0835261e2eb5 None None] MFA Rules not processed for user `37ec649f9c1e4d198e61c583e6a495b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:17.737435 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ef92d0a0-e46e-4811-a337-0835261e2eb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:17.783670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f3cbbf-c1b6-4247-a7ef-d2849a083626 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:17.903828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:17.904460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.904746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.905054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:17.976936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 808/1636] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:17 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:17.978795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:17.980227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:17.982830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:18.073853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:18.079892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:18.107707 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] MFA Rules not processed for user `0e72daa1f1584c47ad375cf8143643b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:18.110983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:18.115317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 829/1637] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:18.117123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:18.140011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d562b4-955e-4ef5-88c9-eb689195bf65 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:18.154994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9529c9-72a3-4f28-849f-733f0859e17a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:18.156407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9529c9-72a3-4f28-849f-733f0859e17a service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:18.157222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9529c9-72a3-4f28-849f-733f0859e17a service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:18.158333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9529c9-72a3-4f28-849f-733f0859e17a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:18.188980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9529c9-72a3-4f28-849f-733f0859e17a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:18.193605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9529c9-72a3-4f28-849f-733f0859e17a service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:18.220717 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 830/1638] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:18 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:18.222349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9cd910-d331-4fc4-aa99-85f6fccf1919 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:18.222645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9cd910-d331-4fc4-aa99-85f6fccf1919 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:18.222978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9cd910-d331-4fc4-aa99-85f6fccf1919 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:18.229794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 831/1639] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 29 02:11:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:18.232080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:18.232413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:18.232907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:18.258086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:18.263245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:18.276858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 832/1640] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:11:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:18.358525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:18.373421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a165278-54cf-4bbc-9f70-43ab48f6b9ae None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:18.381726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:18.383245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:18.384349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:18.384925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:18.404037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:18.406293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 809/1641] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:17 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 429 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:18.407156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:18.436306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 833/1642] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:18.595883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:18.619363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d2acc-8d26-446d-a2fc-93ea84522b4b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:18.623043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:18.623662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:18.623937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:18.624205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:18.643695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:18.646051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:18.679172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 810/1643] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:18 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:18.999075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:19.039219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:19.051686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef33468c-5d42-4408-b9e8-702c1eb6ca78 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:19.064854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e1e58f-e861-4291-bcad-3dda74a8e60c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:19.065588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e1e58f-e861-4291-bcad-3dda74a8e60c service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:19.065872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e1e58f-e861-4291-bcad-3dda74a8e60c service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:19.066142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e1e58f-e861-4291-bcad-3dda74a8e60c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:19.081750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:19.193202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74b1bf0-9eab-47d8-b7d4-5ac9f53002dc service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:19.199569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:19.200376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:19.200727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:19.201558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] PATH_INFO: `/v3/users/0580c757f98549f992eed1e8ff96f896` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:19.203062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0580c757f98549f992eed1e8ff96f896)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:19.204664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: WARNING py.warnings [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:11:19.204664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: warnings.warn(msg) Sep 29 02:11:19.204664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: Sep 29 02:11:19.206538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:19.258727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:19.289270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e1e58f-e861-4291-bcad-3dda74a8e60c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:19.291222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e1e58f-e861-4291-bcad-3dda74a8e60c service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:19.306412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 811/1644] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:19.328633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0580c757f98549f992eed1e8ff96f896'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:19.369282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0580c757f98549f992eed1e8ff96f896'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:19.398114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0580c757f98549f992eed1e8ff96f896'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:19.399023 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0580c757f98549f992eed1e8ff96f896'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:19.401207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0580c757f98549f992eed1e8ff96f896'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:19.427951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 834/1645] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:18 2022] DELETE /identity/v3/users/0580c757f98549f992eed1e8ff96f896 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:19.429496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:19.430774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:19.431215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:19.481131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:19.483528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:19.674571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 812/1646] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:19.691376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:19.693094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:19.833986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a64a0-a459-41ea-b0cb-08da98521919 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:19.841438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe27f6a-c437-4f78-9334-2eb77ae8cbd1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:19.842071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe27f6a-c437-4f78-9334-2eb77ae8cbd1 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:19.842351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe27f6a-c437-4f78-9334-2eb77ae8cbd1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:19.842624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe27f6a-c437-4f78-9334-2eb77ae8cbd1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:19.876277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe27f6a-c437-4f78-9334-2eb77ae8cbd1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:19.878559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe27f6a-c437-4f78-9334-2eb77ae8cbd1 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:19.901050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 835/1647] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:20.218546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:20.220016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:20.238716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c876cbbe-b2bf-4e78-9162-ee696c8fb607 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:20.255866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:20.256537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:20.256819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:20.257123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] PATH_INFO: `/v3/users/cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:20.260155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:20.262343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:20.380309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:20.405254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:20.408248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:20.409277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:20.426683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:20.468135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 813/1648] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:20 2022] DELETE /identity/v3/users/cc4a0c9bb40e4f9eb0ed6f12cb5ebb3e => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:20.473365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:20.474443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:20.474443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:20.535701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:20.537084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:20.643294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:20.668506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 836/1649] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:20.707716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:20.709752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:20.830569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398cd58-989e-400c-b2db-4ba4cea98d85 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:20.839583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11391ca5-7ed7-4534-8de4-7437be9fdc68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:20.840211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11391ca5-7ed7-4534-8de4-7437be9fdc68 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:20.840491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11391ca5-7ed7-4534-8de4-7437be9fdc68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:20.840761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11391ca5-7ed7-4534-8de4-7437be9fdc68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:20.862001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11391ca5-7ed7-4534-8de4-7437be9fdc68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:20.863460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11391ca5-7ed7-4534-8de4-7437be9fdc68 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:20.888239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 814/1650] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:22.629877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:22.631665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:22.652817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa20fa-7144-4d8a-9f5b-4dcc622f1a4e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:22.665209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:22.666459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:22.666751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:22.667022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] PATH_INFO: `/v3/projects/1f133b5228214aa2bbf9b7d5bd35785b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:22.669176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f133b5228214aa2bbf9b7d5bd35785b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:22.671126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:22.805381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f133b5228214aa2bbf9b7d5bd35785b', 'request_id': 'req-0a1655de-b6c4-4664-9150-dd59e382ff34'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:22.822131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f133b5228214aa2bbf9b7d5bd35785b', 'request_id': 'req-0a1655de-b6c4-4664-9150-dd59e382ff34'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:22.830435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f133b5228214aa2bbf9b7d5bd35785b', 'request_id': 'req-0a1655de-b6c4-4664-9150-dd59e382ff34'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:22.856123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] The token cache is being invalidate because project 1f133b5228214aa2bbf9b7d5bd35785b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:22.856565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:22.859993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0a1655de-b6c4-4664-9150-dd59e382ff34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:22.862937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 837/1651] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:22 2022] DELETE /identity/v3/projects/1f133b5228214aa2bbf9b7d5bd35785b => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:22.868210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:22.868740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:22.868740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:22.889970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:22.913210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:22.914491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:23.102271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 815/1652] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:23.109798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.109921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.110194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.144238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:23.145566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:23.168731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 838/1653] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:23.170099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.195761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27877193-81a8-4525-9d0e-95b7a2132210 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.210922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.211893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.212368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.212998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.214056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:23.214600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:23.216663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:23.246474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 816/1654] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:23.250183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.265451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5440c-e2f0-4425-b2be-80f53d6edbb7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.289233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.289639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.289892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.290171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.290830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:23.306672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:23.360366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 839/1655] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:23 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:23.364900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.385760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e9785-0da8-4782-9995-657fe25bceef admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.511175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.513790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:23.515731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.516277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.516646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.517693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.525862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:23.528213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:23.540676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aae844-e0b0-480d-addd-b5210df84355 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.552982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.553856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.554269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.554696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] PATH_INFO: `/v3/projects/62c01a340fd24c00ba44152b74b8ef76` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.559007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=62c01a340fd24c00ba44152b74b8ef76)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:23.561192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:23.589024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 817/1656] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:23 2022] POST /identity/v3/users => generated 287 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:23.590264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.609776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96843f-4214-401c-80f3-04a8f6c616c4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.638253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.639833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.640127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.640901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.641637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:23.644458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:23.666064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 818/1657] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:23 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:23.674889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62c01a340fd24c00ba44152b74b8ef76', 'request_id': 'req-09bf715d-c208-44ca-bda0-32de88fe9f0a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:23.681990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.706657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80868be-37c3-42f5-8df2-d7cb60c05bfd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.706963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62c01a340fd24c00ba44152b74b8ef76', 'request_id': 'req-09bf715d-c208-44ca-bda0-32de88fe9f0a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:23.727893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62c01a340fd24c00ba44152b74b8ef76', 'request_id': 'req-09bf715d-c208-44ca-bda0-32de88fe9f0a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:23.750972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] The token cache is being invalidate because project 62c01a340fd24c00ba44152b74b8ef76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:23.753862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:23.756613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:23.767169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 840/1658] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:23 2022] DELETE /identity/v3/projects/62c01a340fd24c00ba44152b74b8ef76 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:23.768333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.785103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:23.915726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:23.929847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.932720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.933062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.933347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:23.936298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:23.941394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:23.977761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 819/1659] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:23 2022] GET /identity/v3/roles => generated 1420 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:23.978792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:23.978984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bf715d-c208-44ca-bda0-32de88fe9f0a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:23.986036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:23.997293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:23.998295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:23.998295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:23.998496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] PATH_INFO: `/v3/users/37ec649f9c1e4d198e61c583e6a495b9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:24.002021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] RBAC: Authorizing `identity:delete_user(user_id=37ec649f9c1e4d198e61c583e6a495b9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:24.004114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:24.017268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510c5f87-d1e2-4add-94e9-dd05727cdef2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:24.030592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a82d62-9b3c-4a9b-b63c-87fa66530374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:24.031288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a82d62-9b3c-4a9b-b63c-87fa66530374 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:24.031573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a82d62-9b3c-4a9b-b63c-87fa66530374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:24.031852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a82d62-9b3c-4a9b-b63c-87fa66530374 admin admin] PATH_INFO: `/v3/projects/ceba3c7734bb40eaabf7a08839c061c7/users/cd7fec6ad7954139b23617a31ba9dca1/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:24.042049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:24.067853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a82d62-9b3c-4a9b-b63c-87fa66530374 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ceba3c7734bb40eaabf7a08839c061c7, user_id=cd7fec6ad7954139b23617a31ba9dca1, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:24.081211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a82d62-9b3c-4a9b-b63c-87fa66530374 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:24.107228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37ec649f9c1e4d198e61c583e6a495b9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:24.114559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 820/1660] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:23 2022] PUT /identity/v3/projects/ceba3c7734bb40eaabf7a08839c061c7/users/cd7fec6ad7954139b23617a31ba9dca1/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:24.126260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020249e-e040-4b42-9dca-2871096e218d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:24.126889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020249e-e040-4b42-9dca-2871096e218d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:24.127550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020249e-e040-4b42-9dca-2871096e218d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:24.147133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37ec649f9c1e4d198e61c583e6a495b9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:24.173988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37ec649f9c1e4d198e61c583e6a495b9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:24.174774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37ec649f9c1e4d198e61c583e6a495b9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:24.175406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37ec649f9c1e4d198e61c583e6a495b9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:24.192993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e020249e-e040-4b42-9dca-2871096e218d None None] MFA Rules not processed for user `cd7fec6ad7954139b23617a31ba9dca1`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:24.197401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e020249e-e040-4b42-9dca-2871096e218d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:24.206787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 841/1661] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:23 2022] DELETE /identity/v3/users/37ec649f9c1e4d198e61c583e6a495b9 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:24.402995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 821/1662] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:24.404194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:24.406130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:24.634481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec71148-9bc1-4e16-a8f6-18bd5d43110f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:24.643624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:24.644898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:24.645420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:24.645824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:24.646645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:24.650348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:24.694735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 842/1663] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:24 2022] POST /identity/v3/projects => generated 373 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:24.695469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e020249e-e040-4b42-9dca-2871096e218d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:24.709368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e020249e-e040-4b42-9dca-2871096e218d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:24.824536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:24.825495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:24.826223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:24.826223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:24.826995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:24.829241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:24.924312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 822/1664] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:24 2022] POST /identity/v3/users => generated 293 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:11:24.928529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:24.943983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6603a9d-7120-40ec-894b-5f79c970a0ca admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:24.951821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:24.952593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:24.952971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:24.953361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:24.954090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:24.955868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:24.971966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 843/1665] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:24 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:24.979416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:24.998314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46589bc-2192-4158-ad84-570f14515173 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:25.017570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:25.018567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:25.019825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:25.020792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] PATH_INFO: `/v3/projects/51fc6df1d1c0433689a52dd47e976fb3/users/32fe40ffcf5647cdad4ac8bc2e84e7d3/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:25.074284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=51fc6df1d1c0433689a52dd47e976fb3, user_id=32fe40ffcf5647cdad4ac8bc2e84e7d3, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:25.082393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:25.134844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 823/1666] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:24 2022] PUT /identity/v3/projects/51fc6df1d1c0433689a52dd47e976fb3/users/32fe40ffcf5647cdad4ac8bc2e84e7d3/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:25.138957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:25.153681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f3dfd-fc05-4c57-978d-26948c91d346 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:25.195159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:25.274545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4478025e-a484-4cac-a42d-37848f11a804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:25.276541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4478025e-a484-4cac-a42d-37848f11a804 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:25.277440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4478025e-a484-4cac-a42d-37848f11a804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:25.277734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4478025e-a484-4cac-a42d-37848f11a804 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:25.278400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4478025e-a484-4cac-a42d-37848f11a804 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:25.280811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4478025e-a484-4cac-a42d-37848f11a804 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:25.313568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 844/1667] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:25 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:25.318295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:25.345715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d0b0bf-7d05-4ed6-8a0a-70857e4a4bb8 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:25.350921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bb4087-156f-41c9-b445-9fdeb26c4d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:25.352461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bb4087-156f-41c9-b445-9fdeb26c4d31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:25.353351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bb4087-156f-41c9-b445-9fdeb26c4d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:25.353734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bb4087-156f-41c9-b445-9fdeb26c4d31 admin admin] PATH_INFO: `/v3/projects/51fc6df1d1c0433689a52dd47e976fb3/users/32fe40ffcf5647cdad4ac8bc2e84e7d3/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:25.357134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bb4087-156f-41c9-b445-9fdeb26c4d31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=51fc6df1d1c0433689a52dd47e976fb3, user_id=32fe40ffcf5647cdad4ac8bc2e84e7d3, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:25.363790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bb4087-156f-41c9-b445-9fdeb26c4d31 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:25.401923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 824/1668] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:25 2022] PUT /identity/v3/projects/51fc6df1d1c0433689a52dd47e976fb3/users/32fe40ffcf5647cdad4ac8bc2e84e7d3/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:25.413059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:25.413059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:25.417119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:25.451864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:25.452235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:25.452587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:25.518487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] MFA Rules not processed for user `32fe40ffcf5647cdad4ac8bc2e84e7d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:25.521840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:25.553712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] MFA Rules not processed for user `066c7318255e4a01a0f6a81715718e15`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:25.554980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:25.620719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:25.823412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 825/1669] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:25 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 372 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:25.833585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:25.841149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 845/1670] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:25 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:25.856781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:25.858287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:25.861708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:26.013743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887bcd5b-fcec-4433-b747-ee02ed0b4a2c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:26.018618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:26.019420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.020473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.020473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.038529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee8244a-7c39-43e0-8f4a-d61d5dfeb5cd None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:26.044196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:26.046007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:26.058745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ad110-cbc3-4ee4-8e38-58fe09fc55ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:26.059647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ad110-cbc3-4ee4-8e38-58fe09fc55ae service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.061194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ad110-cbc3-4ee4-8e38-58fe09fc55ae service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.062639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ad110-cbc3-4ee4-8e38-58fe09fc55ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.077997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 846/1671] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:25 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:26.083847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ad110-cbc3-4ee4-8e38-58fe09fc55ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:26.087802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ad110-cbc3-4ee4-8e38-58fe09fc55ae service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:26.090862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:26.095917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:26.114762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 826/1672] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:26.240153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:26.287345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f8c961-a145-4d8a-aed1-bd99ad3dde73 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:26.303059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:26.304413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.304698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.304997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] PATH_INFO: `/v3/projects/9a0db52fee0c4d33aef53d9113aab226` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.307353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a0db52fee0c4d33aef53d9113aab226)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:26.313450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:26.453450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a0db52fee0c4d33aef53d9113aab226', 'request_id': 'req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:26.476567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a0db52fee0c4d33aef53d9113aab226', 'request_id': 'req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:26.490927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a0db52fee0c4d33aef53d9113aab226', 'request_id': 'req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:26.532797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] The token cache is being invalidate because project 9a0db52fee0c4d33aef53d9113aab226 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:26.533298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:26.547892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e40327f2-a0d9-46e9-bd55-e814a7ed47ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:26.557616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 847/1673] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:26 2022] DELETE /identity/v3/projects/9a0db52fee0c4d33aef53d9113aab226 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:26.559352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5755701-e219-46d1-a5a1-07bbff42ef0b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.559805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5755701-e219-46d1-a5a1-07bbff42ef0b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.560327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5755701-e219-46d1-a5a1-07bbff42ef0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.622400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b13178-c4f4-4ab6-9e75-9bea4589cfb2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.625983 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b13178-c4f4-4ab6-9e75-9bea4589cfb2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.626538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b13178-c4f4-4ab6-9e75-9bea4589cfb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.635787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d5755701-e219-46d1-a5a1-07bbff42ef0b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:26.648549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d5755701-e219-46d1-a5a1-07bbff42ef0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:26.660575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-32b13178-c4f4-4ab6-9e75-9bea4589cfb2 None None] MFA Rules not processed for user `2ab3ddf040e148789b3c67fc23d2da3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:26.662873 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-32b13178-c4f4-4ab6-9e75-9bea4589cfb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:26.768816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 848/1674] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:26.780608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.780967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.781201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.811422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] MFA Rules not processed for user `2ab3ddf040e148789b3c67fc23d2da3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:26.812842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:26.821939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 849/1675] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:26.837346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:26.839259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:26.916322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 827/1676] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:26.925753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:26.926069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:26.926326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:26.994128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:26.995727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:27.014517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:27.018210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 828/1677] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:27.019320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.023447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:27.051137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303ca24c-fa55-4cfa-9b8c-0728716ffe86 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.065397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.066059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.066342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.066615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] PATH_INFO: `/v3/users/cd7fec6ad7954139b23617a31ba9dca1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.070367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd7fec6ad7954139b23617a31ba9dca1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.072343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.082180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b0a1e7-98b7-4e65-ad52-e5cfa9edb7ac None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.106050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.106695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.106973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.107243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.130595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.133144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.146215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 850/1678] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:26 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:27.147360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.148847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:27.163754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7351568-28a7-4861-8a71-9483e8ecd39c service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.170130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.170389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.170654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.170925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.171831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:27.172134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.173807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.182557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd7fec6ad7954139b23617a31ba9dca1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:27.194844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 851/1679] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:27.195777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.201450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd7fec6ad7954139b23617a31ba9dca1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:27.203071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd7fec6ad7954139b23617a31ba9dca1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:27.206445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd7fec6ad7954139b23617a31ba9dca1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:27.206603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d0557f-fe7f-42ae-a960-5b2c11b5bdf5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.212438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.213584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.213864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.214136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.215172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.217820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.223151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bb707e45-07d5-4e0c-92e4-c3990d04985e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd7fec6ad7954139b23617a31ba9dca1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:27.265212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 852/1680] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:27 2022] POST /identity/v3/projects => generated 357 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:27.268761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.271572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:27.272071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 829/1681] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:27 2022] DELETE /identity/v3/users/cd7fec6ad7954139b23617a31ba9dca1 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:27.279414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.279827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.280111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.325803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:27.329743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:27.392275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1abce-10f9-44a0-908e-ee12c39260bc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.464694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.465227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.465502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.465770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.466407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.468783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.472351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 830/1682] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:27.507863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.508027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 853/1683] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:27 2022] POST /identity/v3/users => generated 279 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:27.518955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a0d85e-a738-4643-943c-e6b39631e28e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.531536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.533697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:27.536429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.537222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.537516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.537793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.538456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.540148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.572080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 831/1684] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:27 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:11:27.576559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.604224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcec3e02-e8bb-4302-8c8d-93e6d218d09c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.618485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.619113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.619398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.619804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.620647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.622689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.650187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 832/1685] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:27 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:27.658124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:27.666311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:27.675604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a52d84-e0e6-4c31-ba59-59a4037a4cc1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.687524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f9ee3-5fc9-454f-89aa-45b9b5a4ed59 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:27.696323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a5536c-0e3f-4dd6-b689-c4d4fef84129 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.696507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:27.697094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.697379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.697656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] PATH_INFO: `/v3/projects/4243388c152446e2877f622683b175ce/users/247c9ef9c44244a9ac51a6b7d249be1f/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.698375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5536c-0e3f-4dd6-b689-c4d4fef84129 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.698753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5536c-0e3f-4dd6-b689-c4d4fef84129 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.699511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5536c-0e3f-4dd6-b689-c4d4fef84129 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.727927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a5536c-0e3f-4dd6-b689-c4d4fef84129 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.731123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4243388c152446e2877f622683b175ce, user_id=247c9ef9c44244a9ac51a6b7d249be1f, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:27.734169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a5536c-0e3f-4dd6-b689-c4d4fef84129 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.734300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:27.758980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 854/1686] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:27.763318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 833/1687] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:27 2022] PUT /identity/v3/projects/4243388c152446e2877f622683b175ce/users/247c9ef9c44244a9ac51a6b7d249be1f/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:27.774339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95599dd3-20d2-45d1-9c77-a574b5049c58 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:27.774596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95599dd3-20d2-45d1-9c77-a574b5049c58 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:27.774876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95599dd3-20d2-45d1-9c77-a574b5049c58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:27.821411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-95599dd3-20d2-45d1-9c77-a574b5049c58 None None] MFA Rules not processed for user `247c9ef9c44244a9ac51a6b7d249be1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:27.824036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-95599dd3-20d2-45d1-9c77-a574b5049c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:27.941440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 855/1688] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:27 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:27.946763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.082405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b405accb-84f4-405e-9d54-820f36cabe30 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.087900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc264e07-b468-4f31-b47a-eda8324b5679 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.088367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc264e07-b468-4f31-b47a-eda8324b5679 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.088810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc264e07-b468-4f31-b47a-eda8324b5679 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.090535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 856/1689] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 29 02:11:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:11:28.101062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.101062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.101062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.109298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.110326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.110608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.110875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.111558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.126182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.143918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:28.198287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 834/1690] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:27 2022] POST /identity/v3/projects => generated 353 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:28.202901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.218505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:28.222107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:28.366757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7651f3-f288-446e-9299-ecde733f3397 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.388583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.389537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.389836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.390116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.390763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.399865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.412904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 857/1691] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:11:28 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:28.463042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 835/1692] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:28 2022] POST /identity/v3/users => generated 283 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:28.464368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.487410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad98cfb-2989-4fc3-835a-45b99c67f3ee None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.492306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.492982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.493267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.493543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.494195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.501214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.513955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.516690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:28.530089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 858/1693] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:28.531547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.538207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c9b35-7975-4259-a298-16b579ffd32d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.547223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.548543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.551559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.551689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414100e9-06e4-46b1-bbd5-030b7bb4b98f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.551987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.563677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.564774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.565742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.566910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] PATH_INFO: `/v3/projects/40e28e104c714cd8a41e0692a1078d7c/users/105b35de0f3b4ddb8c11142fed26e14d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.573200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.581691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.620314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 836/1694] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:28.671044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=40e28e104c714cd8a41e0692a1078d7c, user_id=105b35de0f3b4ddb8c11142fed26e14d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.677203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.697488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 859/1695] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:28 2022] PUT /identity/v3/projects/40e28e104c714cd8a41e0692a1078d7c/users/105b35de0f3b4ddb8c11142fed26e14d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:28.702760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.720274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78fa807-d42b-435a-9efb-1a37fe3029f4 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.881068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098135cb-5ab8-4452-83ef-bcf788225f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.881920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098135cb-5ab8-4452-83ef-bcf788225f82 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.882399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098135cb-5ab8-4452-83ef-bcf788225f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.883010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098135cb-5ab8-4452-83ef-bcf788225f82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.883841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098135cb-5ab8-4452-83ef-bcf788225f82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.889085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098135cb-5ab8-4452-83ef-bcf788225f82 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.909449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 837/1696] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:28.914268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:28.930352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92d198-4098-42b2-bc28-a21affb476dc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:28.949141 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:28.950273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:28.950754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:28.952366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] PATH_INFO: `/v3/projects/40e28e104c714cd8a41e0692a1078d7c/users/105b35de0f3b4ddb8c11142fed26e14d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:28.957347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40e28e104c714cd8a41e0692a1078d7c, user_id=105b35de0f3b4ddb8c11142fed26e14d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:28.967643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:28.998979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 860/1697] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:28 2022] PUT /identity/v3/projects/40e28e104c714cd8a41e0692a1078d7c/users/105b35de0f3b4ddb8c11142fed26e14d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:29.012226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:29.012683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:29.014363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:29.084655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] MFA Rules not processed for user `105b35de0f3b4ddb8c11142fed26e14d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:29.086083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:29.199943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:29.284746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 838/1698] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:29 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:29.300481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:29.302743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:29.385134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:29.461442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fd2270-5216-454a-9459-90035dc2be15 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:29.477747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:29.478587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:29.478865 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:29.479133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:29.502916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:29.504941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:29.530459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 861/1699] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:29 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:29.763335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:29.776267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e5d9c4-b24d-4433-93e6-781da8a1466a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:29.784028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:29.785837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:29.786363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:29.787814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:29.835764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:29.838041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:30.005688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:30.007966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:30.061295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 839/1700] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:29 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:30.067793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1466023c-0a0f-4140-a41d-c0ab7d4d8e5e service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:30.079188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:30.081877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:30.083605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:30.083605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] PATH_INFO: `/v3/users/acfdbe69a1be4bac946a5c99cc9bbc56` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:30.088571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] RBAC: Authorizing `identity:delete_user(user_id=acfdbe69a1be4bac946a5c99cc9bbc56)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:30.094749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:30.215359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:30.217037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:30.316647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'acfdbe69a1be4bac946a5c99cc9bbc56'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:30.386619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'acfdbe69a1be4bac946a5c99cc9bbc56'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:30.420257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'acfdbe69a1be4bac946a5c99cc9bbc56'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:30.422510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'acfdbe69a1be4bac946a5c99cc9bbc56'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:30.427181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'acfdbe69a1be4bac946a5c99cc9bbc56'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:30.507879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 862/1701] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:29 2022] DELETE /identity/v3/users/acfdbe69a1be4bac946a5c99cc9bbc56 => generated 0 bytes in 672 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:30.510120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:30.521673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:30.560187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2df314-a752-45d2-a73b-9845620bbf49 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:30.764195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:30.801955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5b9828-af7c-4ffe-820e-22f8364fd539 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:30.816803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:30.818415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:30.818695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:30.818982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] PATH_INFO: `/v3/projects/ceba3c7734bb40eaabf7a08839c061c7` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:30.820481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ceba3c7734bb40eaabf7a08839c061c7)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:30.822310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:30.824341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:30.824341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:30.824341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:30.826947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:30.828858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:30.857568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:30.859198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd8cfe1-5672-4235-b673-3bcd029871c0 service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:30.889584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 840/1702] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 675 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:30.892998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:30.893288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:30.893574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:30.978525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:30.979974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:30.980089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ceba3c7734bb40eaabf7a08839c061c7', 'request_id': 'req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.018989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ceba3c7734bb40eaabf7a08839c061c7', 'request_id': 'req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.036958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ceba3c7734bb40eaabf7a08839c061c7', 'request_id': 'req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.038215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:31.084435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] The token cache is being invalidate because project ceba3c7734bb40eaabf7a08839c061c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:31.084779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.085579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.090185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 863/1703] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:30 2022] DELETE /identity/v3/projects/ceba3c7734bb40eaabf7a08839c061c7 => generated 0 bytes in 579 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:31.092001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:31.094355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:31.204641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 841/1704] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:31.324437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5262df8-a7d6-49bf-87d8-2ac58d8b3058 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:31.335464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:31.336155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:31.336625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:31.338402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] PATH_INFO: `/v3/users/32fe40ffcf5647cdad4ac8bc2e84e7d3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:31.340054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32fe40ffcf5647cdad4ac8bc2e84e7d3)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:31.358505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:31.403289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:31.404892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:31.618102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffb974a-286c-40c8-9ec7-f08c7e8866cc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:31.627215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32fe40ffcf5647cdad4ac8bc2e84e7d3'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.640227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8871d096-779c-42ce-86bc-31733fcf0d56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:31.641273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871d096-779c-42ce-86bc-31733fcf0d56 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:31.641762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871d096-779c-42ce-86bc-31733fcf0d56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:31.642626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8871d096-779c-42ce-86bc-31733fcf0d56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:31.667774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32fe40ffcf5647cdad4ac8bc2e84e7d3'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.701997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32fe40ffcf5647cdad4ac8bc2e84e7d3'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.703991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32fe40ffcf5647cdad4ac8bc2e84e7d3'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.705865 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32fe40ffcf5647cdad4ac8bc2e84e7d3'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:31.706834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871d096-779c-42ce-86bc-31733fcf0d56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:31.709691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8871d096-779c-42ce-86bc-31733fcf0d56 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:31.728756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:31.755781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 842/1705] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:31.762762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 864/1706] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:31 2022] DELETE /identity/v3/users/32fe40ffcf5647cdad4ac8bc2e84e7d3 => generated 0 bytes in 671 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:33.113640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a08fdf-3d42-462a-ad01-44e9707f94c0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:33.115050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a08fdf-3d42-462a-ad01-44e9707f94c0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:33.115494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a08fdf-3d42-462a-ad01-44e9707f94c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:33.162380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-58a08fdf-3d42-462a-ad01-44e9707f94c0 None None] MFA Rules not processed for user `066c7318255e4a01a0f6a81715718e15`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:33.163832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-58a08fdf-3d42-462a-ad01-44e9707f94c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:33.189382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:33.191828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:33.396063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 843/1707] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:33.415646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363645a-acda-4089-aa1a-034cad51cc36 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:33.422523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:33.423403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:33.424298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:33.424783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] PATH_INFO: `/v3/projects/51fc6df1d1c0433689a52dd47e976fb3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:33.427579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51fc6df1d1c0433689a52dd47e976fb3)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:33.431401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:33.657651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51fc6df1d1c0433689a52dd47e976fb3', 'request_id': 'req-4e1dba76-6e2c-47ba-ac20-90a56d76d318'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:33.686531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51fc6df1d1c0433689a52dd47e976fb3', 'request_id': 'req-4e1dba76-6e2c-47ba-ac20-90a56d76d318'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:33.707164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51fc6df1d1c0433689a52dd47e976fb3', 'request_id': 'req-4e1dba76-6e2c-47ba-ac20-90a56d76d318'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:33.743801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] The token cache is being invalidate because project 51fc6df1d1c0433689a52dd47e976fb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:33.744111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:33.744774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4e1dba76-6e2c-47ba-ac20-90a56d76d318 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:33.753467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 865/1708] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:33 2022] DELETE /identity/v3/projects/51fc6df1d1c0433689a52dd47e976fb3 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:33.757250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:33.757250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:33.757250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:33.790476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:33.793537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:33.965170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 844/1709] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:33.965580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:33.965866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:33.966145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.064525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:34.072082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:34.099898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 866/1710] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:34.103887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:34.115567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:34.120848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f722dba8-0d60-4dea-90ad-6bac81018667 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:34.148639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:34.149403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.150519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.151583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.155205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:34.155780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:34.159017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:34.203355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:34.218711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 845/1711] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:34.227582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ed98b-447c-411c-9eca-c23c8951763d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:34.242140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c2d31-bcec-4891-867c-b489014d936b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:34.243065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c2d31-bcec-4891-867c-b489014d936b admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.243584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c2d31-bcec-4891-867c-b489014d936b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.243924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c2d31-bcec-4891-867c-b489014d936b admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.244921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c2d31-bcec-4891-867c-b489014d936b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:34.248290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c2d31-bcec-4891-867c-b489014d936b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:34.350479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 867/1712] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:34 2022] POST /identity/v3/projects => generated 381 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:11:34.353460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:34.371815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba8c255-c63c-4631-823e-40e41ffdb4c7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:34.507356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.507640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.507911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.552669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:34.553346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.553680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.553949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.555382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:34.560859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:34.604781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] MFA Rules not processed for user `53c020af94d64bac81df7837a86586b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:34.610932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:34.630643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 846/1713] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:34 2022] POST /identity/v3/users => generated 291 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:34.631175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:34.634361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:34.648285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6c3a8-0051-4aad-8f02-9cebf16c1096 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:34.662730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:34.664314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.664818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.666412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] PATH_INFO: `/v3/projects/6329c07d43e4411989cd06fae61982e4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.667813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6329c07d43e4411989cd06fae61982e4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:34.669134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: WARNING py.warnings [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:11:34.669134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: warnings.warn(msg) Sep 29 02:11:34.669134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: Sep 29 02:11:34.671608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:34.767159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:34.803855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6329c07d43e4411989cd06fae61982e4', 'request_id': 'req-b11c3433-77a4-4271-9b95-2a471da9eff1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:34.822393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6329c07d43e4411989cd06fae61982e4', 'request_id': 'req-b11c3433-77a4-4271-9b95-2a471da9eff1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:34.824566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 868/1714] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:34 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:34.826710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:34.841823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a36e49-1a0c-45e4-81bd-c20e1a7d628a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:34.856682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6329c07d43e4411989cd06fae61982e4', 'request_id': 'req-b11c3433-77a4-4271-9b95-2a471da9eff1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:34.887466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] The token cache is being invalidate because project 6329c07d43e4411989cd06fae61982e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:34.887803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:34.890271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b11c3433-77a4-4271-9b95-2a471da9eff1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:34.892803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 847/1715] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:34 2022] DELETE /identity/v3/projects/6329c07d43e4411989cd06fae61982e4 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:34.896595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b739fa6-8db7-471c-9846-c83b17c1234b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.896905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b739fa6-8db7-471c-9846-c83b17c1234b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.897200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b739fa6-8db7-471c-9846-c83b17c1234b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.974876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8b739fa6-8db7-471c-9846-c83b17c1234b None None] MFA Rules not processed for user `f16c1cde815c4a2580cd5ebf7b3ed76d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:34.976266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8b739fa6-8db7-471c-9846-c83b17c1234b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:34.989947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97440af-b395-4bdf-bd85-da4bb2b2e2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:34.990584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97440af-b395-4bdf-bd85-da4bb2b2e2f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:34.990888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97440af-b395-4bdf-bd85-da4bb2b2e2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:34.991153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97440af-b395-4bdf-bd85-da4bb2b2e2f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:34.992190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97440af-b395-4bdf-bd85-da4bb2b2e2f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:34.993446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97440af-b395-4bdf-bd85-da4bb2b2e2f0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:35.038234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 869/1716] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:34 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:35.038234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f316a-e88d-424e-b29f-7a3a8481a0cc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.038234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f316a-e88d-424e-b29f-7a3a8481a0cc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.038234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f316a-e88d-424e-b29f-7a3a8481a0cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.174048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:35.193963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-235f316a-e88d-424e-b29f-7a3a8481a0cc None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:35.196302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-235f316a-e88d-424e-b29f-7a3a8481a0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.203622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 848/1717] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:35.204812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.205398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.205843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.290782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] MFA Rules not processed for user `53c020af94d64bac81df7837a86586b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:35.300275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.464023 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 870/1718] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:11:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 430 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:35.469469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.469469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.469469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.483176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 849/1719] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:35 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:35.492653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:35.495897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.506772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:35.508321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.521355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 871/1720] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:35.525483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:35.535132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.570978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25f3a9-0cda-4e33-81bc-0cfa2c2379d8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:35.592584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:35.593696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.593696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.593850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.594477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:35.596074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:35.625094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 850/1721] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:35 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:35.630892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:35.636662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.708293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:35.792978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c33958-fa95-416c-8d9c-8c3685a3ef3b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:35.814302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb807c1-b450-4b72-ab22-4a1c36e6ea35 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:35.815369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb807c1-b450-4b72-ab22-4a1c36e6ea35 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.815596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb807c1-b450-4b72-ab22-4a1c36e6ea35 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.815876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb807c1-b450-4b72-ab22-4a1c36e6ea35 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.831343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb807c1-b450-4b72-ab22-4a1c36e6ea35 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:35.835819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb807c1-b450-4b72-ab22-4a1c36e6ea35 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:35.839262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e88f7-d445-47d0-a02f-cf9fe0746820 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:35.858352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:35.858994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.859282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.859560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.865012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:35.891328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 872/1722] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:35 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:35.899655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:35.899944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:35.900222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:35.909836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:35.913069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:35.945946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 851/1723] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:35 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:35.955874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:35.992999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:35.994709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:36.005922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664e2ff4-d7ed-476a-a3b9-693a30fc8d37 service nova] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:36.031719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:36.032411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:36.032696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:36.033000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] PATH_INFO: `/v3/projects/08bcf010c9b74308b444e41bc709cb47/users/d143f5986ca942509cdf91e5aeeefb7e/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:36.074190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 873/1724] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:36.077650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:36.087128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:36.103850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=08bcf010c9b74308b444e41bc709cb47, user_id=d143f5986ca942509cdf91e5aeeefb7e, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:36.106069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:36.154368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 852/1725] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:35 2022] PUT /identity/v3/projects/08bcf010c9b74308b444e41bc709cb47/users/d143f5986ca942509cdf91e5aeeefb7e/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:36.166559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:36.360490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:36.418146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb5994-4972-4488-a6af-71df1b3f96ea admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:36.438763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aba16a-5687-4c91-9aeb-a4417e1a765a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:36.443482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:36.445712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:36.451184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:36.451658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:36.453363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:36.453827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:36.455633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311ab74-4ae6-4dcb-88ed-81f7313b19a1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:36.491295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 853/1726] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:36.501771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:36.506247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:36.507302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:36.570290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] MFA Rules not processed for user `d143f5986ca942509cdf91e5aeeefb7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:36.572757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:36.633621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:36.635793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:36.635793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:36.639050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:36.727146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 854/1727] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:36.728362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:36.772293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc38e560-0173-4bd5-8746-fb74fa9557b8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:36.790784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:36.793726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:36.794034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:36.794314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:36.800522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:36.817404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:36.853083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:36.884082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:36.888516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:36.928446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 874/1728] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:11:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 853 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:11:36.929883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:36.931044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 855/1729] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:36 2022] POST /identity/v3/projects => generated 373 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:36.941304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:36.997351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4517aaa2-c9dd-4bc5-9f7f-a4f3dfca792d service swift] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.309721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.310444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.310936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.311286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.316337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.318612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.361689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd5a40-f82c-489b-96f5-9461ad9f9219 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.380674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.386212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.392162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.392529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.402689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.412232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 875/1730] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:36 2022] POST /identity/v3/projects => generated 381 bytes in 483 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:37.417443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.419110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:37.612854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 856/1731] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:36 2022] POST /identity/v3/users => generated 287 bytes in 674 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:37.613202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f38ab-fead-4ef5-bfab-bbc88ba72d8b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.614444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:37.655860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.659786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.660309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.662112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.663024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.666189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.697021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a38163-a400-458b-a733-23ee0c56990c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.706815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 876/1732] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:37 2022] POST /identity/v3/users => generated 297 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:37.708117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:37.721830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158abbd-2265-4d04-b36a-8fcbe080fc87 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.729388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.730004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.730280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.730550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.731430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.734816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.736605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.736810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.737410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.740425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.744798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.759788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.770349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 877/1733] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:37.780411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:37.792579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 857/1734] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:37.796813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:37.821331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdd1aa7-beaf-4122-a73f-2dcaa79cb384 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.823590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c5f742-6dbc-4190-bec1-1dd6eae16455 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.837025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.838002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.838530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.839014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.839885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.842342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e554461-4578-43ff-907e-5f1270add6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.843358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.845174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e554461-4578-43ff-907e-5f1270add6bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.847039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e554461-4578-43ff-907e-5f1270add6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.847532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e554461-4578-43ff-907e-5f1270add6bd admin admin] PATH_INFO: `/v3/projects/d4d122b14d314678a7d25849d94de9a5/users/2607e237043e446ba86a1cf3c67d46cc/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.885213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 858/1735] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:37.890384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:37.911590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc7c001-b216-407c-a65f-5d6893d33923 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:37.929662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:37.930728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:37.931235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:37.931708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] PATH_INFO: `/v3/projects/db715e64f74042749d8c0459127b5711/users/2fa8d7ccbe72467bad5de6cf3607c296/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:37.955059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e554461-4578-43ff-907e-5f1270add6bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4d122b14d314678a7d25849d94de9a5, user_id=2607e237043e446ba86a1cf3c67d46cc, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.964762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e554461-4578-43ff-907e-5f1270add6bd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:37.989071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db715e64f74042749d8c0459127b5711, user_id=2fa8d7ccbe72467bad5de6cf3607c296, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:37.994018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:38.010928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:38.023646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 878/1736] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:37 2022] PUT /identity/v3/projects/d4d122b14d314678a7d25849d94de9a5/users/2607e237043e446ba86a1cf3c67d46cc/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:38.028116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64124b50-22c3-44e4-9312-b0042594a8ae None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:38.028369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64124b50-22c3-44e4-9312-b0042594a8ae None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:38.031455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64124b50-22c3-44e4-9312-b0042594a8ae None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:38.037137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 859/1737] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:37 2022] PUT /identity/v3/projects/db715e64f74042749d8c0459127b5711/users/2fa8d7ccbe72467bad5de6cf3607c296/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:38.038278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:38.049904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 879/1738] 10.222.0.27 () {54 vars in 912 bytes} [Thu Sep 29 02:11:38 2022] GET /identity => generated 270 bytes in 26 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:38.058673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:38.060183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:38.060622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:38.153474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] MFA Rules not processed for user `2fa8d7ccbe72467bad5de6cf3607c296`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:38.157461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:38.244189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6cdd73-6c10-4c20-9a0b-d838b47b7801 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:38.265879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:38.277983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c6ab4f-bd19-4606-a81a-877c9ab751b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:38.278474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c6ab4f-bd19-4606-a81a-877c9ab751b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:38.278709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c6ab4f-bd19-4606-a81a-877c9ab751b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:38.279249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c6ab4f-bd19-4606-a81a-877c9ab751b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:38.286487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c6ab4f-bd19-4606-a81a-877c9ab751b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:38.289660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c6ab4f-bd19-4606-a81a-877c9ab751b2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:38.340392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 860/1739] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:38 2022] GET /identity/v3/roles => generated 1420 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:38.345452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9c7f7-0063-4aec-8533-6afce265c15a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:38.349353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9c7f7-0063-4aec-8533-6afce265c15a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:38.350201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9c7f7-0063-4aec-8533-6afce265c15a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:38.427516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3cc9c7f7-0063-4aec-8533-6afce265c15a None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:38.430328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3cc9c7f7-0063-4aec-8533-6afce265c15a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:38.480262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 880/1740] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:38 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 430 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:38.516261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:38.562721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f5643c-d885-4eb1-b01d-4e1b4323e1d6 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:38.586790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:38.587487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:38.587766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:38.588047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] PATH_INFO: `/v3/projects/d4d122b14d314678a7d25849d94de9a5/users/2607e237043e446ba86a1cf3c67d46cc/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:38.613418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4d122b14d314678a7d25849d94de9a5, user_id=2607e237043e446ba86a1cf3c67d46cc, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:38.623011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:38.682850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 881/1741] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:38 2022] PUT /identity/v3/projects/d4d122b14d314678a7d25849d94de9a5/users/2607e237043e446ba86a1cf3c67d46cc/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:38.688526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:38.717012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 861/1742] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:11:38 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 373 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:38.718764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:38.719415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:38.719882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:38.817279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] MFA Rules not processed for user `2607e237043e446ba86a1cf3c67d46cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:38.819482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:38.929411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 862/1743] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:38 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:38.930796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:38.950885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:39.003714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925ae351-91d7-4ea3-a0e6-90ec17938034 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:39.006133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:39.033031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdde6d9-77a5-49c4-921b-6d88a932e65a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:39.033664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdde6d9-77a5-49c4-921b-6d88a932e65a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:39.033941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdde6d9-77a5-49c4-921b-6d88a932e65a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:39.034217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdde6d9-77a5-49c4-921b-6d88a932e65a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:39.156358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdde6d9-77a5-49c4-921b-6d88a932e65a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:39.157830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032f79f7-fac3-4a85-b82c-b8eef9b51b0e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:39.162162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdde6d9-77a5-49c4-921b-6d88a932e65a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:39.166865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:39.167856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:39.168418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:39.168945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:39.182566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 882/1744] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:38 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:39.183814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:39.184075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:39.184348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:39.218183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:39.219993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:39.232453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:39.233730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:39.251793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 883/1745] 10.222.0.57 () {58 vars in 1015 bytes} [Thu Sep 29 02:11:39 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:39.252786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:39.256186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 863/1746] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:39.256728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:39.342842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:39.344493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:39.351557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:39.386805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d13ae-5ef6-41a6-b2c9-d77033c07873 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:39.388019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5116bfb-b1d7-4939-9417-87c813f0d3db None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:39.397052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:39.410273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:39.410518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:39.410636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:39.410906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:39.411044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:39.411343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:39.411733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:39.421792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:39.423664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:39.436193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 864/1747] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:39.467074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:39.474196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:39.503958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 884/1748] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:39 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:40.792677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:40.809826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ca8c59-1d06-4527-aea9-31ae9b7bbbcc service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:40.816666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2607e237043e446ba86a1cf3c67d46cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd4d122b14d314678a7d25849d94de9a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:40.817645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:40.817999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:40.818359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:40.819309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:40.821131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:40.836097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:40.849738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 865/1749] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:40.854728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:40.871461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e1f57e-a78b-4345-8734-cd62f74a93b8 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:40.879666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2607e237043e446ba86a1cf3c67d46cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd4d122b14d314678a7d25849d94de9a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:40.882067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:40.882769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:40.883642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:40.885910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:40.889524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:40.977366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 885/1750] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:40 2022] POST /identity/v3/projects => generated 354 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:11:40.980508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:40.997822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd35e646-88c7-45f9-8e20-5a2479902472 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:41.210008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2607e237043e446ba86a1cf3c67d46cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd4d122b14d314678a7d25849d94de9a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:41.232065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:41.232406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:41.232700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:41.233514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:41.233825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:41.251374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c6dc84-6be9-4c51-850c-209169463582 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:41.306811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 866/1751] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:41.561587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:41.583741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1c3adc-d194-47b1-8a96-e4b3d1fb0d62 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:41.621349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2607e237043e446ba86a1cf3c67d46cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd4d122b14d314678a7d25849d94de9a5', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:41.623136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:41.623425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:41.623837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] PATH_INFO: `/v3/projects/baffdf3b34044376857dd591af959aff` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:41.633736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorizing `identity:delete_project(project_id=baffdf3b34044376857dd591af959aff)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:41.643161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:41.819734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'baffdf3b34044376857dd591af959aff', 'request_id': 'req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:41.835665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'baffdf3b34044376857dd591af959aff', 'request_id': 'req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:41.854658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'baffdf3b34044376857dd591af959aff', 'request_id': 'req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:41.940465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] The token cache is being invalidate because project baffdf3b34044376857dd591af959aff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:41.940465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:41.940465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:41.940465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 886/1752] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:41 2022] DELETE /identity/v3/projects/baffdf3b34044376857dd591af959aff => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:42.676962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e211338d-97e9-43be-a1b4-fff6419bccad None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:42.678576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e211338d-97e9-43be-a1b4-fff6419bccad None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:42.679024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e211338d-97e9-43be-a1b4-fff6419bccad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:42.681151 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:42.683694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:42.738643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e211338d-97e9-43be-a1b4-fff6419bccad None None] MFA Rules not processed for user `53c020af94d64bac81df7837a86586b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:42.741259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e211338d-97e9-43be-a1b4-fff6419bccad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:42.949446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 867/1753] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:42 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:42.953325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc01c2c-1b55-4cf2-b032-ba9b4968e9a0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:42.954232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc01c2c-1b55-4cf2-b032-ba9b4968e9a0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:42.955089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc01c2c-1b55-4cf2-b032-ba9b4968e9a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:43.004849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7bc01c2c-1b55-4cf2-b032-ba9b4968e9a0 None None] MFA Rules not processed for user `f16c1cde815c4a2580cd5ebf7b3ed76d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:43.012790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7bc01c2c-1b55-4cf2-b032-ba9b4968e9a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:43.079208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea98f3e2-93f3-41c4-a25c-8e86b0126ad9 tempest-VolumeQuotaClassesTest-2018174810 tempest-VolumeQuotaClassesTest-2018174810-project-admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:43.109501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:43.114187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:43.125617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:43.134835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:43.135210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] PATH_INFO: `/v3/users/d143f5986ca942509cdf91e5aeeefb7e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:43.143632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=d143f5986ca942509cdf91e5aeeefb7e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:43.171445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:43.269382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 868/1754] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:43.273573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d2f20c-957a-4c29-89b1-479da33b1873 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:43.273867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d2f20c-957a-4c29-89b1-479da33b1873 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:43.274148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d2f20c-957a-4c29-89b1-479da33b1873 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:43.289149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 869/1755] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:11:43 2022] GET /identity => generated 270 bytes in 19 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:43.428365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:43.430168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd143f5986ca942509cdf91e5aeeefb7e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:43.517582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd143f5986ca942509cdf91e5aeeefb7e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:43.561501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd143f5986ca942509cdf91e5aeeefb7e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:43.568549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd143f5986ca942509cdf91e5aeeefb7e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:43.572093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-642f9245-866b-490d-a948-c7812d4986ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd143f5986ca942509cdf91e5aeeefb7e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:43.635106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501702a-9382-468f-936d-976cb29630c3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:43.635458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501702a-9382-468f-936d-976cb29630c3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:43.637693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501702a-9382-468f-936d-976cb29630c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:43.666722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 887/1756] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:42 2022] DELETE /identity/v3/users/d143f5986ca942509cdf91e5aeeefb7e => generated 0 bytes in 986 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:43.674210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:43.685326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:43.688124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:43.695942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d501702a-9382-468f-936d-976cb29630c3 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:43.698258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d501702a-9382-468f-936d-976cb29630c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:43.851423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:43.859433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:44.047483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 870/1757] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:11:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 414 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:44.143453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 888/1758] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 470 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:44.317897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501702a-9382-468f-936d-976cb29630c3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:44.319660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d501702a-9382-468f-936d-976cb29630c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:44.632989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501702a-9382-468f-936d-976cb29630c3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:44.636959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcbaddc-f7bc-47d6-8a0a-25acacccf644 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:44.637466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbaddc-f7bc-47d6-8a0a-25acacccf644 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:44.637740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbaddc-f7bc-47d6-8a0a-25acacccf644 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:44.638015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbaddc-f7bc-47d6-8a0a-25acacccf644 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:44.657346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcbaddc-f7bc-47d6-8a0a-25acacccf644 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:44.658972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcbaddc-f7bc-47d6-8a0a-25acacccf644 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:44.700965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 871/1759] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:45.421287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:45.422591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:45.454107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69107fe7-1347-4841-945e-54e1a9f4960e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:45.476607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:45.478920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:45.481969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:45.483477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] PATH_INFO: `/v3/users/2fa8d7ccbe72467bad5de6cf3607c296` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:45.487728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fa8d7ccbe72467bad5de6cf3607c296)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:45.500032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:45.656939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fa8d7ccbe72467bad5de6cf3607c296'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:45.699378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fa8d7ccbe72467bad5de6cf3607c296'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:45.712061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fa8d7ccbe72467bad5de6cf3607c296'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:45.713405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fa8d7ccbe72467bad5de6cf3607c296'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:45.714345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fa8d7ccbe72467bad5de6cf3607c296'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:45.756377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 889/1760] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:45 2022] DELETE /identity/v3/users/2fa8d7ccbe72467bad5de6cf3607c296 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:45.757434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:45.758291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:45.758571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:45.808116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:45.809902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:45.976465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 872/1761] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:45.998133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:46.000137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:46.128858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:46.213765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfce142-c932-41e3-b670-a9eadd0016ab admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:46.218268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:46.219176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:46.219698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:46.220047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:46.234223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:46.244562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:46.262665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 890/1762] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:46.720510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:46.724141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:46.768671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5f6a3-f0c3-4e73-a57b-82128bd4c3e9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:46.785639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:46.796556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:46.797523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:46.798241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] PATH_INFO: `/v3/projects/08bcf010c9b74308b444e41bc709cb47` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:46.801997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=08bcf010c9b74308b444e41bc709cb47)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:46.810067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:46.837426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:46.838934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:46.856582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:46.950624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08bcf010c9b74308b444e41bc709cb47', 'request_id': 'req-881febce-8a5a-473b-8348-50497d9f3c5d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:46.971353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08bcf010c9b74308b444e41bc709cb47', 'request_id': 'req-881febce-8a5a-473b-8348-50497d9f3c5d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:46.986946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08bcf010c9b74308b444e41bc709cb47', 'request_id': 'req-881febce-8a5a-473b-8348-50497d9f3c5d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.017175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] The token cache is being invalidate because project 08bcf010c9b74308b444e41bc709cb47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:47.018461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.019361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.023527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 873/1763] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:46 2022] DELETE /identity/v3/projects/08bcf010c9b74308b444e41bc709cb47 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:47.032649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:47.038300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:47.071497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c6d7e5-6e7a-4014-8a0d-97cbc3e77918 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:47.229338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881febce-8a5a-473b-8348-50497d9f3c5d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:47.244553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:47.248409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:47.249437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:47.251617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] PATH_INFO: `/v3/users/2607e237043e446ba86a1cf3c67d46cc` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:47.256494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2607e237043e446ba86a1cf3c67d46cc)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:47.260228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:47.261171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:47.261926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:47.263335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:47.263849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:47.268669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:47.298726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:47.300752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda8da07-1c39-4288-a1fa-db4efd7fd59a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:47.334041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 891/1764] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:47.375986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2607e237043e446ba86a1cf3c67d46cc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.399085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2607e237043e446ba86a1cf3c67d46cc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.401415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2607e237043e446ba86a1cf3c67d46cc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.403608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2607e237043e446ba86a1cf3c67d46cc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.423623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1d40ab7-d919-4701-9eb6-8788eefdb6fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2607e237043e446ba86a1cf3c67d46cc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:47.524428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 874/1765] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:11:47 2022] DELETE /identity/v3/users/2607e237043e446ba86a1cf3c67d46cc => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:48.010476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10773834-36d4-452b-939a-b4199132017a None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:48.011203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10773834-36d4-452b-939a-b4199132017a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:48.011203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10773834-36d4-452b-939a-b4199132017a None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:48.012402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 892/1766] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:11:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:48.119752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8aeb6c-d250-4a27-ad0e-f5bd2ef883c1 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:48.120332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8aeb6c-d250-4a27-ad0e-f5bd2ef883c1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:48.120332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8aeb6c-d250-4a27-ad0e-f5bd2ef883c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:48.200580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8f8aeb6c-d250-4a27-ad0e-f5bd2ef883c1 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:48.200580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8f8aeb6c-d250-4a27-ad0e-f5bd2ef883c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:48.254163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:48.442205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10773834-36d4-452b-939a-b4199132017a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:48.443988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-10773834-36d4-452b-939a-b4199132017a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:48.464495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 875/1767] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:11:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:48.688458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10773834-36d4-452b-939a-b4199132017a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:48.710828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:48.712494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:48.712991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:48.713475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] PATH_INFO: `/v3/projects/db715e64f74042749d8c0459127b5711` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:48.715131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db715e64f74042749d8c0459127b5711)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:48.717205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:48.719470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:48.908305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db715e64f74042749d8c0459127b5711', 'request_id': 'req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:48.956162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db715e64f74042749d8c0459127b5711', 'request_id': 'req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:48.987089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db715e64f74042749d8c0459127b5711', 'request_id': 'req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:49.062925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] The token cache is being invalidate because project db715e64f74042749d8c0459127b5711 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:49.069891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:49.077576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:49.092310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 893/1768] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:48 2022] DELETE /identity/v3/projects/db715e64f74042749d8c0459127b5711 => generated 0 bytes in 651 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:49.097126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c276d26-dd29-4e19-aa8c-2b61277e3798 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:49.097725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c276d26-dd29-4e19-aa8c-2b61277e3798 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:49.098292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c276d26-dd29-4e19-aa8c-2b61277e3798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:49.198346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5c276d26-dd29-4e19-aa8c-2b61277e3798 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:49.209576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5c276d26-dd29-4e19-aa8c-2b61277e3798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:49.262212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:49.264337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:49.531702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 876/1769] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:49.540028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:49.543236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:49.545988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:49.612452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bd9c67-6a60-4a0c-b3d6-ce7ce87cb4e9 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:49.651647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:49.654961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:49.657473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:49.658975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:49.659439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:49.660166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:49.667946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:49.689274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 877/1770] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:49.694419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:49.716098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:49.720123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:49.730658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7624db-997c-4f6d-9cff-a2707865dd87 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:49.761408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:49.762278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:49.762797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:49.763286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:49.764186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:49.764849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:49.767655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:49.788544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 894/1771] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:49.810427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 878/1772] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:49.815328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:49.852907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b3f52a-14c5-46d7-bd71-71004b5696c0 service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:49.873375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:49.874324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:49.874864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:49.875413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:49.877102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:49.879549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:50.007590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 895/1773] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:49 2022] POST /identity/v3/projects => generated 377 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:50.012159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:50.036523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91d373-7a57-40c6-8590-418fe9663f55 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:50.301460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:50.302084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:50.302360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:50.302637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:50.311518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:50.315868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:50.366860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 879/1774] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:50 2022] POST /identity/v3/users => generated 289 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:50.371567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:50.399948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9421a089-e140-4786-a8a1-12d7362833f1 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:50.410708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:50.412796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:50.414307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:50.414786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:50.417073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:50.425341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:50.469875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 896/1775] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:50.471517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:50.490319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d217aefd-ca29-4ead-b350-cde7bd198cea admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:50.510887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b6a46f-454d-4790-90d1-17efd6d5ab8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:50.534716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b6a46f-454d-4790-90d1-17efd6d5ab8e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:50.537306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b6a46f-454d-4790-90d1-17efd6d5ab8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:50.538286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b6a46f-454d-4790-90d1-17efd6d5ab8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:50.540408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b6a46f-454d-4790-90d1-17efd6d5ab8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:50.544662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b6a46f-454d-4790-90d1-17efd6d5ab8e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:50.591208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 880/1776] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:50.594647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:50.611203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ce4ead-91c5-4622-a44f-6622e872ef5d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:50.633710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:50.634646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:50.635166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:50.635579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] PATH_INFO: `/v3/projects/51872300f3994c7d859ce74a47015d09/users/31a10e99dbdc493da395f9a1cf63270f/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:50.680801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=51872300f3994c7d859ce74a47015d09, user_id=31a10e99dbdc493da395f9a1cf63270f, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:50.689071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:50.726364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 897/1777] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:50 2022] PUT /identity/v3/projects/51872300f3994c7d859ce74a47015d09/users/31a10e99dbdc493da395f9a1cf63270f/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:50.739480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:50.739774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:50.740046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:50.903128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:50.914074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] MFA Rules not processed for user `31a10e99dbdc493da395f9a1cf63270f`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:50.918281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:51.096682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.099809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:51.113001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 881/1778] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:50 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:51.118045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.139893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:51.234238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01ce3a5-8e5d-41f0-ad41-99a13d7f26c7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.246367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.247572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.248100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.248384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] PATH_INFO: `/v3/projects/d4d122b14d314678a7d25849d94de9a5` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.250923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4d122b14d314678a7d25849d94de9a5)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.254543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.302884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c799b6-0526-4e5a-832c-759cedd70021 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.314533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.315317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.315424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.315694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.318152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.321711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.365899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4d122b14d314678a7d25849d94de9a5', 'request_id': 'req-e2099535-6579-4ea4-8af7-1873ec2cf00a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:51.381635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 882/1779] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/projects => generated 377 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:51.386017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.387141 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4d122b14d314678a7d25849d94de9a5', 'request_id': 'req-e2099535-6579-4ea4-8af7-1873ec2cf00a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:51.398072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4d122b14d314678a7d25849d94de9a5', 'request_id': 'req-e2099535-6579-4ea4-8af7-1873ec2cf00a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:51.422389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] The token cache is being invalidate because project d4d122b14d314678a7d25849d94de9a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:11:51.422690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:51.426265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e2099535-6579-4ea4-8af7-1873ec2cf00a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:11:51.433294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 898/1780] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:11:51 2022] DELETE /identity/v3/projects/d4d122b14d314678a7d25849d94de9a5 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:51.435183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe31ddb-cacd-46c7-a81e-6ccd1873b6aa None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.435753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe31ddb-cacd-46c7-a81e-6ccd1873b6aa None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.436167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe31ddb-cacd-46c7-a81e-6ccd1873b6aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.462478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ffe31ddb-cacd-46c7-a81e-6ccd1873b6aa None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:51.463855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ffe31ddb-cacd-46c7-a81e-6ccd1873b6aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:51.509584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0238a-bc32-42ae-85f1-42719957ff79 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.511240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.511931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.512288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.512655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.513394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.515419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.538536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 883/1781] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/users => generated 295 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:51.542881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.544809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:51.561175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8a8f4-5f82-4244-809d-128fb18f7bb3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.567546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.568282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.568761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.569194 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.570039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.578732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 899/1782] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:51.578825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.581497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.581784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.583999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 884/1783] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:51.584563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.589402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.605166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43062276-b098-4956-9043-0618c2b9b202 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.613596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.614283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.614560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.614953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] PATH_INFO: `/v3/projects/7b7b3e86685347cbb73d9e80036b45fa/users/1aef935d95ac40ca82cddf1a709ff671/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.617475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:51.629315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:51.630766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:51.642382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 900/1784] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:51.646468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.647107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b7b3e86685347cbb73d9e80036b45fa, user_id=1aef935d95ac40ca82cddf1a709ff671, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.651077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.658001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c131349-e7d1-4e7f-93b7-a5c8fd59f716 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.664240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.665346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.665628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.666201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.666914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:11:51.667216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.668921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.691765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 901/1785] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:11:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:11:51.693426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.705160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 885/1786] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:51 2022] PUT /identity/v3/projects/7b7b3e86685347cbb73d9e80036b45fa/users/1aef935d95ac40ca82cddf1a709ff671/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:51.706174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.724734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f35eec-9a40-4f12-8250-3ddf1c6c1227 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.760778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:51.780817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f960b5a1-de29-48e3-a7d0-8b351f10d53a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.786486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.787492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.787917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.788333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.789155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.790955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.805879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.806579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.806926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.807264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.809360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.811484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.812284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 886/1787] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:51.813588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.825192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60b6e38-5a13-4003-8b86-f0f16ea757fb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.830857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de331f52-d392-4e03-85e6-07cf7669b15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.831542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de331f52-d392-4e03-85e6-07cf7669b15d admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.832059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de331f52-d392-4e03-85e6-07cf7669b15d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.832159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de331f52-d392-4e03-85e6-07cf7669b15d admin admin] PATH_INFO: `/v3/projects/7b7b3e86685347cbb73d9e80036b45fa/users/1aef935d95ac40ca82cddf1a709ff671/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.834230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de331f52-d392-4e03-85e6-07cf7669b15d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b7b3e86685347cbb73d9e80036b45fa, user_id=1aef935d95ac40ca82cddf1a709ff671, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.837082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de331f52-d392-4e03-85e6-07cf7669b15d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.853006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 902/1788] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/projects => generated 401 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:11:51.856955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 887/1789] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:51 2022] PUT /identity/v3/projects/7b7b3e86685347cbb73d9e80036b45fa/users/1aef935d95ac40ca82cddf1a709ff671/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:51.860259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.868456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.868764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.869048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.903742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] MFA Rules not processed for user `1aef935d95ac40ca82cddf1a709ff671`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:51.905437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:51.946904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c97b4d2-fec9-4e0c-a54b-ae32d4cf845e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.951720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.952326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.952605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.952895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.953536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:51.955457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:51.977690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 903/1790] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/users => generated 301 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:51.981811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.987646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 888/1791] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:51 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:51.992539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add07338-98bf-4dd3-855c-e7345a4dc055 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:51.997427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:51.998038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:51.998311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:51.998583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:51.998990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:51.999223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.000553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:52.000812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.015653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 904/1792] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:11:52.016682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.027771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff1bfe6-0ae7-43e1-b5e8-ef5b393c887f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.034196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.035261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.035549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.036112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.036746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.040348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.054597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 905/1793] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:52 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:52.058663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.069018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8a707-48d6-4875-a6e8-ed62df31cb10 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.072004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1721e9-f96c-4a27-af40-a16dd9b5dc0d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.073911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.074672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.074993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.075402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.077560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:52.077682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c7ea50-d5f2-4bd1-ac9b-e5c43891259b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.078330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7ea50-d5f2-4bd1-ac9b-e5c43891259b admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.078621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7ea50-d5f2-4bd1-ac9b-e5c43891259b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.078898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7ea50-d5f2-4bd1-ac9b-e5c43891259b admin admin] PATH_INFO: `/v3/projects/763a4434805e4d12b655a8c6a8ef14f4/users/e7266a0fff164627996194fade272126/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.083299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:52.094561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c7ea50-d5f2-4bd1-ac9b-e5c43891259b admin admin] RBAC: Authorizing `identity:create_grant(project_id=763a4434805e4d12b655a8c6a8ef14f4, user_id=e7266a0fff164627996194fade272126, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.097335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c7ea50-d5f2-4bd1-ac9b-e5c43891259b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.123378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 906/1794] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:52 2022] PUT /identity/v3/projects/763a4434805e4d12b655a8c6a8ef14f4/users/e7266a0fff164627996194fade272126/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:52.134779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.135433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.137050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.149709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.151144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.160394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 889/1795] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:51 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:52.172948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] MFA Rules not processed for user `e7266a0fff164627996194fade272126`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:52.174174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:52.228896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:52.257229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 907/1796] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:52 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:52.258310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.275150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a13318-33ce-421b-8e50-73cf300f388e service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.354324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.354973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.355228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.355501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.356146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.358077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.388908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 890/1797] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:11:52 2022] POST /identity/v3/projects => generated 401 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:52.390185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.401227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ba6c34-027e-4637-8565-1626be9c379a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.493482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.493482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.493482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.493852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.494322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.496402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.533352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 908/1798] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:11:52 2022] POST /identity/v3/users => generated 307 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:11:52.538125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.549851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7346dbca-ca51-455e-9700-38a54a4f3035 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.555487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.556188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.556537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.556928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.557650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.559383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.579050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 891/1799] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:52 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:11:52.587711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.602956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f00454-254a-464b-a4fe-1569d00de2b8 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.608086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.608741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.609055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.609330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] PATH_INFO: `/v3/projects/6f2c67a62fe046b888c2bbd88e890d03/users/b83d23fbed704589ba6b1b7062619096/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.624918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f2c67a62fe046b888c2bbd88e890d03, user_id=b83d23fbed704589ba6b1b7062619096, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.627582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.645574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 909/1800] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:52 2022] PUT /identity/v3/projects/6f2c67a62fe046b888c2bbd88e890d03/users/b83d23fbed704589ba6b1b7062619096/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:11:52.649592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.659735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4cb11-3e52-466a-9a93-3ed2401c0289 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.714285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8c4cf7-665d-4aaa-8cd7-318d2266ea92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.714882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8c4cf7-665d-4aaa-8cd7-318d2266ea92 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.715160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8c4cf7-665d-4aaa-8cd7-318d2266ea92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.715430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8c4cf7-665d-4aaa-8cd7-318d2266ea92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.716055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8c4cf7-665d-4aaa-8cd7-318d2266ea92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.717657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8c4cf7-665d-4aaa-8cd7-318d2266ea92 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.727594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 892/1801] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:11:52 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:11:52.731814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.742432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ce856-2056-4acb-8bf6-db7ea3f99bf0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.747049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.747692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.747967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.748235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] PATH_INFO: `/v3/projects/6f2c67a62fe046b888c2bbd88e890d03/users/b83d23fbed704589ba6b1b7062619096/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.750155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f2c67a62fe046b888c2bbd88e890d03, user_id=b83d23fbed704589ba6b1b7062619096, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.752777 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.763775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 910/1802] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:11:52 2022] PUT /identity/v3/projects/6f2c67a62fe046b888c2bbd88e890d03/users/b83d23fbed704589ba6b1b7062619096/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:11:52.774245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.774539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.774819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.802366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] MFA Rules not processed for user `b83d23fbed704589ba6b1b7062619096`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:52.803513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:52.807406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:52.873048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 893/1803] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:11:52 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:52.883496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:52.885219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:52.940657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58754c8b-3378-4290-8c41-fac88eb5e140 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:52.944290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:52.944910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:52.945191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:52.945467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:52.954638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:52.956093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:52.966155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 911/1804] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:52 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:53.043549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:53.053769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa1e78-b60d-46c7-9ca1-55388827c783 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:53.057354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:53.057954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:53.058236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:53.058513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:53.115346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:53.116790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:53.126248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 894/1805] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:53 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:00.099948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:00.108800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea40066-23f1-445a-ad0e-f934940a7af9 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:00.113145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf909a2-2bb4-4428-b53a-3584c40b10e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:00.113739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf909a2-2bb4-4428-b53a-3584c40b10e1 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:00.114017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf909a2-2bb4-4428-b53a-3584c40b10e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:00.114287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf909a2-2bb4-4428-b53a-3584c40b10e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:00.156989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:00.178513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf909a2-2bb4-4428-b53a-3584c40b10e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:00.180005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf909a2-2bb4-4428-b53a-3584c40b10e1 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:00.190415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 912/1806] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:00 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:00.730076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:00.816363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b50884-efce-4770-9a1a-e558b9b1c4d0 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:00.822757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:00.823646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:00.824119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:00.824582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] PATH_INFO: `/v3/users/e7266a0fff164627996194fade272126` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:00.826145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7266a0fff164627996194fade272126)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:00.828516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:00.887554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:00.895404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7266a0fff164627996194fade272126'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:00.906314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7266a0fff164627996194fade272126'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:00.908218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7266a0fff164627996194fade272126'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:00.908218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7266a0fff164627996194fade272126'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:00.922955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7266a0fff164627996194fade272126'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:00.956768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 895/1807] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:00 2022] DELETE /identity/v3/users/e7266a0fff164627996194fade272126 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:00.961145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:00.961445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:00.961841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:00.993902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:00.995422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.053789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:01.055251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.148526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 913/1808] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:01.153901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d41a6d5-f53a-4894-b3b2-4688ff465c9c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:01.158985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:01.159450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:01.160605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:01.160605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] PATH_INFO: `/v3/users/247c9ef9c44244a9ac51a6b7d249be1f` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:01.161943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=247c9ef9c44244a9ac51a6b7d249be1f)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:01.164119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:01.170560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:01.173967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.233517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '247c9ef9c44244a9ac51a6b7d249be1f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.257603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '247c9ef9c44244a9ac51a6b7d249be1f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.259691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '247c9ef9c44244a9ac51a6b7d249be1f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.261520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '247c9ef9c44244a9ac51a6b7d249be1f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.282060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f75a15b6-ceba-43e4-8ec1-771396b0faf4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '247c9ef9c44244a9ac51a6b7d249be1f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.287595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79be8cf1-e573-4424-aeac-57419e54d757 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:01.293278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:01.294475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:01.295081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:01.295543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:01.298305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.322358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:01.323582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:01.333292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 896/1809] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:01 2022] DELETE /identity/v3/users/247c9ef9c44244a9ac51a6b7d249be1f => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:01.338082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da117180-8582-464e-8b99-9c5c8377097b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:01.338490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da117180-8582-464e-8b99-9c5c8377097b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:01.338772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da117180-8582-464e-8b99-9c5c8377097b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:01.342992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 914/1810] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:01.366890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-da117180-8582-464e-8b99-9c5c8377097b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:01.368147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da117180-8582-464e-8b99-9c5c8377097b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.445095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:01.493176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 897/1811] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:01.507625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:01.512660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.530798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12eeaeb8-7081-44a4-afea-804327125391 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:01.608610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:01.609417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:01.609795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:01.610172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:01.621300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:01.622842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:01.634321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 915/1812] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:01.857048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da117180-8582-464e-8b99-9c5c8377097b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:01.858530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da117180-8582-464e-8b99-9c5c8377097b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:01.872422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da117180-8582-464e-8b99-9c5c8377097b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:01.877874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:01.878862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:01.879210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:01.879594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] PATH_INFO: `/v3/projects/763a4434805e4d12b655a8c6a8ef14f4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:01.880839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=763a4434805e4d12b655a8c6a8ef14f4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:01.882894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:01.933214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '763a4434805e4d12b655a8c6a8ef14f4', 'request_id': 'req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.941018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '763a4434805e4d12b655a8c6a8ef14f4', 'request_id': 'req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.947587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '763a4434805e4d12b655a8c6a8ef14f4', 'request_id': 'req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.959433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] The token cache is being invalidate because project 763a4434805e4d12b655a8c6a8ef14f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:01.959720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.960382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:01.962109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 898/1813] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:01 2022] DELETE /identity/v3/projects/763a4434805e4d12b655a8c6a8ef14f4 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:01.965886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:01.975741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a352d-6849-4f86-b4e2-dc79bfeb4f8c service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:01.996569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:02.001843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:02.003682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.044792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:02.045671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.045952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.046233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] PATH_INFO: `/v3/users/b83d23fbed704589ba6b1b7062619096` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.047368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b83d23fbed704589ba6b1b7062619096)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:02.049408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:02.095575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1d73e-6a2b-4ea1-891f-328cafcd2bcf admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:02.096559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:02.097446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.097732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.098007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] PATH_INFO: `/v3/projects/4243388c152446e2877f622683b175ce` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.099236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4243388c152446e2877f622683b175ce)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:02.101242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:02.114192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b83d23fbed704589ba6b1b7062619096'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.127924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b83d23fbed704589ba6b1b7062619096'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.135681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b83d23fbed704589ba6b1b7062619096'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.136380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b83d23fbed704589ba6b1b7062619096'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.137099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b83d23fbed704589ba6b1b7062619096'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.153254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 916/1814] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:01 2022] DELETE /identity/v3/users/b83d23fbed704589ba6b1b7062619096 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:02.155548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4243388c152446e2877f622683b175ce', 'request_id': 'req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.168392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4243388c152446e2877f622683b175ce', 'request_id': 'req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.175092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4243388c152446e2877f622683b175ce', 'request_id': 'req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.177246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:02.188945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] The token cache is being invalidate because project 4243388c152446e2877f622683b175ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:02.189270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.190215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.192335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 899/1815] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:02 2022] DELETE /identity/v3/projects/4243388c152446e2877f622683b175ce => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:02.196500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:02.197970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.264657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159da722-bc55-48e5-a94c-a1efb5dc0866 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:02.270040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:02.271380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.272008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.272386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] PATH_INFO: `/v3/users/105b35de0f3b4ddb8c11142fed26e14d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.273728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=105b35de0f3b4ddb8c11142fed26e14d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:02.276737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:02.359165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:02.363989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '105b35de0f3b4ddb8c11142fed26e14d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.389415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '105b35de0f3b4ddb8c11142fed26e14d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.397101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '105b35de0f3b4ddb8c11142fed26e14d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.397954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '105b35de0f3b4ddb8c11142fed26e14d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.398651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '105b35de0f3b4ddb8c11142fed26e14d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.416569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 917/1816] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:02 2022] DELETE /identity/v3/users/105b35de0f3b4ddb8c11142fed26e14d => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:02.481001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:02.482721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.650870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7e8e6f-3da0-4d8d-a7b1-68bb87e241e4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:02.723629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:02.724531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.724531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.724818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] PATH_INFO: `/v3/projects/6f2c67a62fe046b888c2bbd88e890d03` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.726020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f2c67a62fe046b888c2bbd88e890d03)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:02.727935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:02.766255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:02.768073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.789583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d777b7d-f0a5-4c38-9d9f-895b8e25eb7a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:02.803644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:02.804455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.804739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.805062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] PATH_INFO: `/v3/projects/40e28e104c714cd8a41e0692a1078d7c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.806339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40e28e104c714cd8a41e0692a1078d7c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:02.808310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:02.812338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f2c67a62fe046b888c2bbd88e890d03', 'request_id': 'req-7a3c1280-9627-4ede-940c-9673c74a502c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.821033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f2c67a62fe046b888c2bbd88e890d03', 'request_id': 'req-7a3c1280-9627-4ede-940c-9673c74a502c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.827479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f2c67a62fe046b888c2bbd88e890d03', 'request_id': 'req-7a3c1280-9627-4ede-940c-9673c74a502c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.839169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] The token cache is being invalidate because project 6f2c67a62fe046b888c2bbd88e890d03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:02.839517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.840261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a3c1280-9627-4ede-940c-9673c74a502c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.841819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 900/1817] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:02 2022] DELETE /identity/v3/projects/6f2c67a62fe046b888c2bbd88e890d03 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:02.846651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d1d02-6ed0-41ca-b621-185e8082cf1d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.846935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d1d02-6ed0-41ca-b621-185e8082cf1d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.847770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d1d02-6ed0-41ca-b621-185e8082cf1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.861167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40e28e104c714cd8a41e0692a1078d7c', 'request_id': 'req-0e2de320-a788-4aa6-9341-a4924ff47f39'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.867767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3d2d1d02-6ed0-41ca-b621-185e8082cf1d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:02.869178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3d2d1d02-6ed0-41ca-b621-185e8082cf1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.869294 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40e28e104c714cd8a41e0692a1078d7c', 'request_id': 'req-0e2de320-a788-4aa6-9341-a4924ff47f39'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.875075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40e28e104c714cd8a41e0692a1078d7c', 'request_id': 'req-0e2de320-a788-4aa6-9341-a4924ff47f39'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.886459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] The token cache is being invalidate because project 40e28e104c714cd8a41e0692a1078d7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:02.886749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.887389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0e2de320-a788-4aa6-9341-a4924ff47f39 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:02.888891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 918/1818] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:02 2022] DELETE /identity/v3/projects/40e28e104c714cd8a41e0692a1078d7c => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:02.893587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0362a-dc2a-49e5-a1ff-b30098aca768 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.893950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0362a-dc2a-49e5-a1ff-b30098aca768 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.894318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0362a-dc2a-49e5-a1ff-b30098aca768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.894781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:02.914304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-42b0362a-dc2a-49e5-a1ff-b30098aca768 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:02.915628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-42b0362a-dc2a-49e5-a1ff-b30098aca768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.930481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 901/1819] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:02.939440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.940066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.940349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.959965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:02.961211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:02.969258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 902/1820] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:02.974186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:02.974794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 919/1821] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:02.982370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.982659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.982931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.983374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9942ffe9-2a5b-480f-8a85-108df2464826 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:02.988034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:02.988623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:02.988928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:02.989204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:02.989838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:02.990140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:02.991749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.001747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:03.002924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:03.007058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 903/1822] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:12:03.007913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.011402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 920/1823] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:03.015306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.017350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42edfe2e-b082-4ff6-aa36-436a992d69bc admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.022458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.023123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.023479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.023822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.024078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8b28de-3d04-4455-9d8d-5f821f0b37d7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.024546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.026531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.028715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.029324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.029599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.029865 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.030492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:03.030790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.032349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.044204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 921/1824] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:12:03.048759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.054614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 904/1825] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:03.058480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.058810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3885c672-33e3-4946-8237-60de5a80c349 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.079382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:03.119744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.121025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.121375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.121716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.122414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.123985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44a219b-dc15-4245-bae8-57f507684ebe admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.124362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.129220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.129832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.130115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.130376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.131017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.132918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.157051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 905/1826] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/users => generated 287 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:03.157229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 922/1827] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/projects => generated 367 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:03.161079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.162049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.175216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e783830-47ec-443f-8ba4-3dde8a8d7471 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.229451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf98872-eec5-49de-a3b5-0b286af902ac admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.234468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.235078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.235359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.235633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.235725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.236269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.236368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.236597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.236907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.237543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.238282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.239178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.249295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 906/1828] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:03.253273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.262782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 923/1829] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/users => generated 284 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:03.264309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e161421c-2e6f-42a9-afb7-49f1d64717b4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.266631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.269626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.270267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.270551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.270826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.271469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.273097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.277753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0a4a9-5d3f-4416-8c6f-65b4c23ce26c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.282472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.283061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.283349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.283628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.283742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 907/1830] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:03.284266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.285952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.287943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.293142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:03.296903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 924/1831] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:03.299290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14262b79-35b1-4922-a0e2-49fbbede6260 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.301096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.306438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f85a026-84fe-406f-9c31-9c19519210b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.307139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f85a026-84fe-406f-9c31-9c19519210b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.307691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f85a026-84fe-406f-9c31-9c19519210b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.307970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f85a026-84fe-406f-9c31-9c19519210b8 admin admin] PATH_INFO: `/v3/projects/c1827a761e194efd89ba99913eb0063f/users/5e5d53ce354a4e0294e3f226d685a3e5/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.310977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba8ff00-2957-4a28-8624-d542cf681d5d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.316123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.316824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.317154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.317427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.318055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.319677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.323043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f85a026-84fe-406f-9c31-9c19519210b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1827a761e194efd89ba99913eb0063f, user_id=5e5d53ce354a4e0294e3f226d685a3e5, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.325709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f85a026-84fe-406f-9c31-9c19519210b8 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.330121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 925/1832] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:03.334936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.343166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 908/1833] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:03 2022] PUT /identity/v3/projects/c1827a761e194efd89ba99913eb0063f/users/5e5d53ce354a4e0294e3f226d685a3e5/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:03.344947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84f762c-7091-45ab-ac9d-f6943b7ffd19 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.353355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.353646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.353925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.381678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] MFA Rules not processed for user `5e5d53ce354a4e0294e3f226d685a3e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:03.382909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:03.401858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821a78d7-c8b9-44f3-be8f-28db1a20b87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.402613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a78d7-c8b9-44f3-be8f-28db1a20b87c admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.402962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a78d7-c8b9-44f3-be8f-28db1a20b87c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.403242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a78d7-c8b9-44f3-be8f-28db1a20b87c admin admin] PATH_INFO: `/v3/projects/4163240905384e438796918a306f2f8d/users/6663b7e4b47843caa0940773ce8a96cd/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.418630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821a78d7-c8b9-44f3-be8f-28db1a20b87c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4163240905384e438796918a306f2f8d, user_id=6663b7e4b47843caa0940773ce8a96cd, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.421331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821a78d7-c8b9-44f3-be8f-28db1a20b87c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.431604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:03.436325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 926/1834] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:03 2022] PUT /identity/v3/projects/4163240905384e438796918a306f2f8d/users/6663b7e4b47843caa0940773ce8a96cd/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:03.448104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.448808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.448808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.459537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 909/1835] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:03.473025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.474506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:03.478088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] MFA Rules not processed for user `6663b7e4b47843caa0940773ce8a96cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:03.479269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:03.548237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911368c3-670e-4751-8af0-6408d8acf712 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.563007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 927/1836] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:03.570298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.611919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.612557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.614493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.614493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.679495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:03.701610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce86374-52f0-4dc9-bf91-401f0869203a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.715047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.715747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.716095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.716375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.717402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.720200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.726938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.729273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.752030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 910/1837] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:03.776723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.811881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cbc2c3-45d8-4e83-b43f-96126785e33f service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.822236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 928/1838] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/projects => generated 367 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:03.827347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.827928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.829100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.829482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.829858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] PATH_INFO: `/v3/users/5e5d53ce354a4e0294e3f226d685a3e5` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.831135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e5d53ce354a4e0294e3f226d685a3e5)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.833799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.919661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e5d53ce354a4e0294e3f226d685a3e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:03.924430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be07441-8c0d-4ff2-99b0-28b5ab1bee75 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:03.929607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e5d53ce354a4e0294e3f226d685a3e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:03.930320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e5d53ce354a4e0294e3f226d685a3e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:03.930567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:03.931094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e5d53ce354a4e0294e3f226d685a3e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:03.931339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.931688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.932403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.932941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:03.935155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:03.943875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-37bc7b6c-26ee-4846-bc1c-a83f59a66041 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e5d53ce354a4e0294e3f226d685a3e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:03.970162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 929/1839] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/users => generated 290 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:03.974917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 911/1840] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:03 2022] DELETE /identity/v3/users/5e5d53ce354a4e0294e3f226d685a3e5 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:03.977791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:03.979826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:03.980329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:03.980811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:03.981119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:03.996097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:04.003099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e5de7a-82f8-4b47-9153-06dc88350492 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:04.013797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:04.015038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:04.124374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:04.125352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.125931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.127224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:04.128346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:04.133835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:04.138582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 912/1841] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:04.161004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 930/1842] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:04.165667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:04.174860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:04.179232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:04.180856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f9304e-dc7a-47d8-90d7-f74e92aed793 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:04.199906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:04.201888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.202014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.202351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] PATH_INFO: `/v3/projects/77960357ceea445288a4ba929875fa1e/users/d91d0e9562484218b26294e9ff151458/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:04.236554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77960357ceea445288a4ba929875fa1e, user_id=d91d0e9562484218b26294e9ff151458, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:04.247197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:04.296349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 913/1843] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:04 2022] PUT /identity/v3/projects/77960357ceea445288a4ba929875fa1e/users/d91d0e9562484218b26294e9ff151458/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:04.297310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:04.373282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:04.375614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca1202b-0e1d-4bb4-bb3e-8fb5f6ce869b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:04.469121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cfc885-ce6c-4cfe-9ee5-e7911b1e542f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:04.469909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cfc885-ce6c-4cfe-9ee5-e7911b1e542f service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.470257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cfc885-ce6c-4cfe-9ee5-e7911b1e542f service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.470661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cfc885-ce6c-4cfe-9ee5-e7911b1e542f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:04.475607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e87185-1f13-4199-9621-012d4c988a32 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:04.482492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cfc885-ce6c-4cfe-9ee5-e7911b1e542f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:04.488389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:04.489611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.489901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.492916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:04.493336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:04.495129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:04.497901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cfc885-ce6c-4cfe-9ee5-e7911b1e542f service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:04.521506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 914/1844] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:04 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:04.521847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 931/1845] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:04.525934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:04.545440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4deb067e-b593-487f-bced-61e14debe09c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:04.556903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:04.558091 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.558373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.558713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] PATH_INFO: `/v3/projects/77960357ceea445288a4ba929875fa1e/users/d91d0e9562484218b26294e9ff151458/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:04.564660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=77960357ceea445288a4ba929875fa1e, user_id=d91d0e9562484218b26294e9ff151458, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:04.571939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:04.606529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 915/1846] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:04 2022] PUT /identity/v3/projects/77960357ceea445288a4ba929875fa1e/users/d91d0e9562484218b26294e9ff151458/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:04.618323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.619168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.620072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:04.673980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] MFA Rules not processed for user `d91d0e9562484218b26294e9ff151458`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:04.675415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:04.762394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:04.810345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 932/1847] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:04 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:04.823979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:04.826227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:04.960381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd42cb53-7092-4730-a0a8-0a606d670a9a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:04.968762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b74923-0d1e-4301-9a80-21f0d89b4ab9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:04.969742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b74923-0d1e-4301-9a80-21f0d89b4ab9 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:04.970181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b74923-0d1e-4301-9a80-21f0d89b4ab9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:04.970490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b74923-0d1e-4301-9a80-21f0d89b4ab9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:05.013695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b74923-0d1e-4301-9a80-21f0d89b4ab9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:05.015691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b74923-0d1e-4301-9a80-21f0d89b4ab9 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:05.030720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 916/1848] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:04 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:06.153133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:06.155132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:06.342030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542ca6d0-3b0f-4ec3-8651-e0a4700c1821 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:06.357198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:06.357973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.358264 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.358543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] PATH_INFO: `/v3/projects/c1827a761e194efd89ba99913eb0063f` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.359758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1827a761e194efd89ba99913eb0063f)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:06.361752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:06.469701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8923d2-52ea-4479-911a-e4ed0b8bed09 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.470143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8923d2-52ea-4479-911a-e4ed0b8bed09 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.470279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8923d2-52ea-4479-911a-e4ed0b8bed09 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.471600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 917/1849] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:12:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:06.526976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.527257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.527537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.531430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1827a761e194efd89ba99913eb0063f', 'request_id': 'req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:06.550586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1827a761e194efd89ba99913eb0063f', 'request_id': 'req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:06.567490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1827a761e194efd89ba99913eb0063f', 'request_id': 'req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:06.591559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] The token cache is being invalidate because project c1827a761e194efd89ba99913eb0063f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:06.591877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:06.592553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-9eb2bfbe-7144-45ac-a74e-8b4d2a8f3744 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:06.599190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 933/1850] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:06 2022] DELETE /identity/v3/projects/c1827a761e194efd89ba99913eb0063f => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:06.602125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc52f99-eeb1-4a93-baca-9e01979c5a60 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.604372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc52f99-eeb1-4a93-baca-9e01979c5a60 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.605154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc52f99-eeb1-4a93-baca-9e01979c5a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.613646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:06.631007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:06.642242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0cc52f99-eeb1-4a93-baca-9e01979c5a60 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:06.643911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0cc52f99-eeb1-4a93-baca-9e01979c5a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:06.754589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 934/1851] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:06.771594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.772720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.774928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.836249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:06.837226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:06.871811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 918/1852] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:12:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:06.872915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 935/1853] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:06.877357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:06.894659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb26c9b7-634c-426e-b467-ccd3ec7b378b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:06.903716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:06.904344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.904677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.904995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.906249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:06.906249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:06.907914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:06.931297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 919/1854] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:06.932782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:06.944109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b6d3b9-ada0-48db-84af-c0e167e52013 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:06.956581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:06.957238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:06.957518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:06.957786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:06.958498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:06.960334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:06.974540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:07.022557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.024790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:07.040977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 936/1855] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:06 2022] POST /identity/v3/projects => generated 375 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:07.045785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.060278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2151900f-7ffa-4920-b2de-e35b500f4aea admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.150503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:07.172285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c620b-b27f-4479-bf3e-7620f97c514b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.180727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.181449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.181728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.182002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.182637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.184546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.202140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.202856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.203143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.203420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.213846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.217618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 937/1856] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:07 2022] POST /identity/v3/users => generated 288 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:07.217757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.218713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.237484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 920/1857] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:07.239204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a0fd6a-5d68-482b-aef2-023178a6d347 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.244252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.244924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.245194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.245518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.246177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.247789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.262856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 938/1858] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:07 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:07.264144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.287850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13647e85-dee4-40a3-be9c-c895836829ed service nova] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.302509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54817ef-e718-456d-a4a9-326696024374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.303358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54817ef-e718-456d-a4a9-326696024374 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.303891 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54817ef-e718-456d-a4a9-326696024374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.304077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54817ef-e718-456d-a4a9-326696024374 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.304694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54817ef-e718-456d-a4a9-326696024374 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.306361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54817ef-e718-456d-a4a9-326696024374 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.333155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 921/1859] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:07 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:07.337213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.348296 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b967c8-5dff-4d59-9664-720e91c79230 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.358101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.359214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.359747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.360268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] PATH_INFO: `/v3/projects/4c6dfa04d2f24ac68cfbf3f544d30797/users/408ae9f6dd9245c4b1e707e0c5bc44df/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.382501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c6dfa04d2f24ac68cfbf3f544d30797, user_id=408ae9f6dd9245c4b1e707e0c5bc44df, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.385530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.402614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 939/1860] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:07 2022] PUT /identity/v3/projects/4c6dfa04d2f24ac68cfbf3f544d30797/users/408ae9f6dd9245c4b1e707e0c5bc44df/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:07.414131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.414412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.414683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.446856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] MFA Rules not processed for user `408ae9f6dd9245c4b1e707e0c5bc44df`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:07.448599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:07.568152 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.570324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 922/1861] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:07 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:07.573870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:07.581941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.585675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:07.660904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cc6375-90af-4e85-a10d-53c84ca245d4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.665813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e06159b-a5d4-4377-8207-0804fc68a946 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.666038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e06159b-a5d4-4377-8207-0804fc68a946 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.666605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e06159b-a5d4-4377-8207-0804fc68a946 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.666978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e06159b-a5d4-4377-8207-0804fc68a946 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.676020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51daa128-34c2-4598-87a9-3720c81bad27 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:07.677689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e06159b-a5d4-4377-8207-0804fc68a946 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.681383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e06159b-a5d4-4377-8207-0804fc68a946 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.688310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:07.689742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.691198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.691905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] PATH_INFO: `/v3/users/6663b7e4b47843caa0940773ce8a96cd` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.693931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 923/1862] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:07.696729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=6663b7e4b47843caa0940773ce8a96cd)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:07.703156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:07.759786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:07.762473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6663b7e4b47843caa0940773ce8a96cd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:07.781406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6663b7e4b47843caa0940773ce8a96cd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:07.789366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6663b7e4b47843caa0940773ce8a96cd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:07.790822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6663b7e4b47843caa0940773ce8a96cd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:07.791949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6663b7e4b47843caa0940773ce8a96cd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:07.807209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 940/1863] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:07 2022] DELETE /identity/v3/users/6663b7e4b47843caa0940773ce8a96cd => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:07.812128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:07.812537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:07.812800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:07.834296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:07.835817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:07.878237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:07.912916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 924/1864] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:07.922164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:07.923630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:08.000084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab93f53-36c4-4de0-a51d-c47c333306cc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:08.004074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:08.004890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:08.005242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:08.005583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:08.016148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:08.017833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:08.040937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 941/1865] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:08.512793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:08.514863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:08.527272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be67bbd9-0c92-40f7-a36d-046eb07f78af None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:08.532119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:08.532755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:08.533066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:08.533347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] PATH_INFO: `/v3/projects/4163240905384e438796918a306f2f8d` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:08.535062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4163240905384e438796918a306f2f8d)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:08.537005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:08.587475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4163240905384e438796918a306f2f8d', 'request_id': 'req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.597704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4163240905384e438796918a306f2f8d', 'request_id': 'req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.604554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4163240905384e438796918a306f2f8d', 'request_id': 'req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.617967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] The token cache is being invalidate because project 4163240905384e438796918a306f2f8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:08.618592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.619278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.620999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 925/1866] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:08 2022] DELETE /identity/v3/projects/4163240905384e438796918a306f2f8d => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:08.624963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:08.637200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51792a46-3cef-4ebf-917a-84d57774e87e service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:08.708718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:08.709298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:08.709578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:08.709850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] PATH_INFO: `/v3/users/d91d0e9562484218b26294e9ff151458` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:08.710959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d91d0e9562484218b26294e9ff151458)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:08.712904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:08.770597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd91d0e9562484218b26294e9ff151458'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.782938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd91d0e9562484218b26294e9ff151458'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.784138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:08.793711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd91d0e9562484218b26294e9ff151458'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.794533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd91d0e9562484218b26294e9ff151458'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.795318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-63792fb3-08f8-45ae-ba61-3102b7349fd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd91d0e9562484218b26294e9ff151458'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:08.810240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 942/1867] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:08 2022] DELETE /identity/v3/users/d91d0e9562484218b26294e9ff151458 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:09.084711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.086180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:09.102233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb2687-60c9-46ec-ae1b-edaf7dce4e85 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.161480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.162114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.162399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.162672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] PATH_INFO: `/v3/projects/77960357ceea445288a4ba929875fa1e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.163784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77960357ceea445288a4ba929875fa1e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.165735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.212305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77960357ceea445288a4ba929875fa1e', 'request_id': 'req-b97b5821-a814-4704-a82c-de9199faa558'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:09.219877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77960357ceea445288a4ba929875fa1e', 'request_id': 'req-b97b5821-a814-4704-a82c-de9199faa558'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:09.226667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77960357ceea445288a4ba929875fa1e', 'request_id': 'req-b97b5821-a814-4704-a82c-de9199faa558'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:09.228648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:09.239361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] The token cache is being invalidate because project 77960357ceea445288a4ba929875fa1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:09.239733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:09.240423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b97b5821-a814-4704-a82c-de9199faa558 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:09.241940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 926/1868] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:09 2022] DELETE /identity/v3/projects/77960357ceea445288a4ba929875fa1e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:09.246867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.247161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.247436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.266931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:09.268130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:09.328386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 943/1869] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:09.336359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.336649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.337103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.363207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:09.364843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:09.376423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 927/1870] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:09.377855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.388118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ad52c0-bcdf-4ff6-8224-e5f237738bba None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.392634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.393320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.393603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.393916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.394565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:09.394986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.396566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.408571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 944/1871] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:09.413055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.422903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4462246-5f2d-410f-8954-bfbd98089eed None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.428115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.428733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.429039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.429310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.429945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.431838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.463922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 928/1872] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:09 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:09.465056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.475536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82de5cd-f993-480e-a011-7ef5df294eaa admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.541040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.541646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.541925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.542199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.542821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.544732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.565868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 945/1873] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:09 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:09.570204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.580360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f1130-23d7-4afe-995e-673eea687280 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.585494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.586096 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.586375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.586656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.587320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.588958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.599708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 929/1874] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:09.604241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.614729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf9ab1a-3ced-41d6-a077-855915e01d26 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.619455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80c9307-d9c2-4d73-947b-c2e62a36dd6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.620054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c9307-d9c2-4d73-947b-c2e62a36dd6d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.620339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c9307-d9c2-4d73-947b-c2e62a36dd6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.620614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80c9307-d9c2-4d73-947b-c2e62a36dd6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.621329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c9307-d9c2-4d73-947b-c2e62a36dd6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.622927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80c9307-d9c2-4d73-947b-c2e62a36dd6d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.633884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 946/1875] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:09.638022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.647873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f42aeb-65c6-4be9-8df4-9d20069929ef admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.652510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.653185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.653476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.653746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] PATH_INFO: `/v3/projects/140961b8a1874f3aa0b560a7557f9b9c/users/9495379f1712440cad3cbdbb09523101/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.668605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=140961b8a1874f3aa0b560a7557f9b9c, user_id=9495379f1712440cad3cbdbb09523101, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.671289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:09.684988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 930/1876] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:09 2022] PUT /identity/v3/projects/140961b8a1874f3aa0b560a7557f9b9c/users/9495379f1712440cad3cbdbb09523101/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:09.696402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.696687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.696995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.726637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] MFA Rules not processed for user `9495379f1712440cad3cbdbb09523101`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:09.727837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:09.779476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:09.816570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 947/1877] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:09.821555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:09.926212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99efd3ea-9833-4835-a9f5-62228c4d6ec1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:09.933077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:09.934151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:09.934622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:09.934962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:09.936179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:09.938805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.005122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 931/1878] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:09 2022] POST /identity/v3/projects => generated 385 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:10.009309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.025732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ae88fb-0631-48b5-938b-197bdffdc788 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.099635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.100261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.100555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.100817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.101493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.103430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.133989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 948/1879] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:10 2022] POST /identity/v3/users => generated 299 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:10.138336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.155648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e393d32-0f0c-46bd-a6b9-e07af09f1707 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.162342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.163089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.163520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.163893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.164648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.166661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.183757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 932/1880] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:10.193106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.207226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae6fe1a-721f-4b0e-b764-17a2ebe16fd5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.222221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.223240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.223607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.224216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] PATH_INFO: `/v3/projects/9f9b9ff3140545f781e2bdefbb70ff7a/users/2a9b91c2f1aa4441b688d26852a145f2/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.247451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f9b9ff3140545f781e2bdefbb70ff7a, user_id=2a9b91c2f1aa4441b688d26852a145f2, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.250349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.271488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 949/1881] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:10 2022] PUT /identity/v3/projects/9f9b9ff3140545f781e2bdefbb70ff7a/users/2a9b91c2f1aa4441b688d26852a145f2/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:10.275521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.287005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03754159-dd90-4c95-a5da-7161fcaedf4b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.405416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d717ca5d-4059-4163-a0f0-5253fe0c3700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.406064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717ca5d-4059-4163-a0f0-5253fe0c3700 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.406342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717ca5d-4059-4163-a0f0-5253fe0c3700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.406614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d717ca5d-4059-4163-a0f0-5253fe0c3700 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.407251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717ca5d-4059-4163-a0f0-5253fe0c3700 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.409251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d717ca5d-4059-4163-a0f0-5253fe0c3700 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.411559 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:10.425584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 933/1882] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:10.426670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.436824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8e474f-935e-4e14-905e-2f49db8152c6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.442458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.443113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.443386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.443670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] PATH_INFO: `/v3/projects/9f9b9ff3140545f781e2bdefbb70ff7a/users/2a9b91c2f1aa4441b688d26852a145f2/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.445566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f9b9ff3140545f781e2bdefbb70ff7a, user_id=2a9b91c2f1aa4441b688d26852a145f2, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.448193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.459454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 950/1883] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:10 2022] PUT /identity/v3/projects/9f9b9ff3140545f781e2bdefbb70ff7a/users/2a9b91c2f1aa4441b688d26852a145f2/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:10.469941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.470233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.470510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.499583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] MFA Rules not processed for user `2a9b91c2f1aa4441b688d26852a145f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:10.500794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:10.584979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 934/1884] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:10 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:10.598760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.600519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:10.669760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd8628-7f12-4393-a258-70c618a027bb admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.673533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5968ef5a-80dc-4099-b55b-031c183d4d8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.674277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5968ef5a-80dc-4099-b55b-031c183d4d8b service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.674631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5968ef5a-80dc-4099-b55b-031c183d4d8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.675029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5968ef5a-80dc-4099-b55b-031c183d4d8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.685533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5968ef5a-80dc-4099-b55b-031c183d4d8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.687040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5968ef5a-80dc-4099-b55b-031c183d4d8b service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.698146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 951/1885] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:10 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:10.813149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:10.883118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ef350e-4694-41c4-ad3a-59c2fcc18e36 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:10.887723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:10.888361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.888642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.888954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] PATH_INFO: `/v3/users/9495379f1712440cad3cbdbb09523101` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.890147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9495379f1712440cad3cbdbb09523101)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:10.892099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:10.939126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9495379f1712440cad3cbdbb09523101'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:10.946207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9495379f1712440cad3cbdbb09523101'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:10.946859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9495379f1712440cad3cbdbb09523101'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:10.947486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9495379f1712440cad3cbdbb09523101'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:10.954650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9495379f1712440cad3cbdbb09523101'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:10.971691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 935/1886] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:10 2022] DELETE /identity/v3/users/9495379f1712440cad3cbdbb09523101 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:10.976783 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:10.977101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:10.977378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:10.996641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:10.997525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:11.036832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:11.076373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 952/1887] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:11.087575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:11.089085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:11.096495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:11.161827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7063d49a-276c-4e83-94d9-73b09dcd646e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:11.165474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:11.166213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:11.166475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:11.166751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:11.176502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:11.177950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:11.188016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 936/1888] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:11.481984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:11.483497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:11.495772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d19addc-0014-4530-8da7-a1e29cec98b2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:11.500524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:11.501219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:11.501498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:11.501771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] PATH_INFO: `/v3/projects/140961b8a1874f3aa0b560a7557f9b9c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:11.502912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=140961b8a1874f3aa0b560a7557f9b9c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:11.504860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:11.551369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '140961b8a1874f3aa0b560a7557f9b9c', 'request_id': 'req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.559056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '140961b8a1874f3aa0b560a7557f9b9c', 'request_id': 'req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.565536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '140961b8a1874f3aa0b560a7557f9b9c', 'request_id': 'req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.578584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] The token cache is being invalidate because project 140961b8a1874f3aa0b560a7557f9b9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:11.578879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.579519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.581213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 953/1889] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:11 2022] DELETE /identity/v3/projects/140961b8a1874f3aa0b560a7557f9b9c => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:11.586817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:11.595020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426c48e0-dd87-4e25-ba5f-7d831c4fcb79 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:11.651335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:11.651966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:11.652247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:11.652520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] PATH_INFO: `/v3/users/2a9b91c2f1aa4441b688d26852a145f2` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:11.653679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a9b91c2f1aa4441b688d26852a145f2)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:11.655626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:11.700674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a9b91c2f1aa4441b688d26852a145f2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.707420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a9b91c2f1aa4441b688d26852a145f2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.708076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a9b91c2f1aa4441b688d26852a145f2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.708679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a9b91c2f1aa4441b688d26852a145f2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.716184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1d3563a3-4619-4b1b-8018-90c15b5dc128 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a9b91c2f1aa4441b688d26852a145f2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:11.733780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 937/1890] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:11 2022] DELETE /identity/v3/users/2a9b91c2f1aa4441b688d26852a145f2 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:12.240915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.243896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:12.260380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3c2132-17e7-43cf-a096-4dd1ac98edf5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.345237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.346006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.346358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.346697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] PATH_INFO: `/v3/projects/9f9b9ff3140545f781e2bdefbb70ff7a` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.348045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f9b9ff3140545f781e2bdefbb70ff7a)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.350148 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.402837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f9b9ff3140545f781e2bdefbb70ff7a', 'request_id': 'req-3df9f919-0b7b-4294-b610-29999c81e57b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:12.411087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f9b9ff3140545f781e2bdefbb70ff7a', 'request_id': 'req-3df9f919-0b7b-4294-b610-29999c81e57b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:12.413315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:12.419277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f9b9ff3140545f781e2bdefbb70ff7a', 'request_id': 'req-3df9f919-0b7b-4294-b610-29999c81e57b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:12.432120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] The token cache is being invalidate because project 9f9b9ff3140545f781e2bdefbb70ff7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:12.432436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:12.433254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-3df9f919-0b7b-4294-b610-29999c81e57b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:12.434748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 954/1891] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:12 2022] DELETE /identity/v3/projects/9f9b9ff3140545f781e2bdefbb70ff7a => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:12.439460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.440169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.440276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.460350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:12.461589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:12.480236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:12.522233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 938/1892] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:12.530272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.530561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.530840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.551952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:12.553403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:12.561517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 955/1893] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:12.566164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.576252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b0d865-8585-4040-88e9-da366a1c8625 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.582451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.583538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.583538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.583780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.584395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:12.584704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.587491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.601369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 939/1894] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:12:12.605270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.614790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3952a7b-934b-409b-bdde-11f0bc519d8f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.620554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.621216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.621497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.621772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.622441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.624352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.659761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 956/1895] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:12 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:12.661569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.671855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411b522a-23ff-49db-82aa-b27dedc7bd00 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.729003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.729597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.729884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.730155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.730826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.732663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.755603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 940/1896] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:12 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:12.759845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.770109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cb942-4c32-43db-beb9-8f5754b0df4d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.774772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.775369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.775653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.775925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.776567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.778268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.788532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 957/1897] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:12.792452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.802969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a340ee4d-7678-4242-a88f-0ff219be8d6b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.807806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d5ef78-c459-47d1-984c-617c215ed75f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.808395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5ef78-c459-47d1-984c-617c215ed75f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.808679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5ef78-c459-47d1-984c-617c215ed75f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.808987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5ef78-c459-47d1-984c-617c215ed75f admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.809623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d5ef78-c459-47d1-984c-617c215ed75f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.811196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d5ef78-c459-47d1-984c-617c215ed75f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.825012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 941/1898] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:12.829482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:12.839905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca56dd4e-24d7-4325-82da-30660655e0c7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:12.844568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:12.845245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.845525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.845798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] PATH_INFO: `/v3/projects/57f2d6b240a847e7aa1c748a6464005b/users/51b9642eccfa4f7bbf5362489763a2af/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.860644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57f2d6b240a847e7aa1c748a6464005b, user_id=51b9642eccfa4f7bbf5362489763a2af, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:12.863885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:12.876631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 958/1899] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:12 2022] PUT /identity/v3/projects/57f2d6b240a847e7aa1c748a6464005b/users/51b9642eccfa4f7bbf5362489763a2af/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:12.887619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:12.887903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:12.888174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:12.915308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] MFA Rules not processed for user `51b9642eccfa4f7bbf5362489763a2af`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:12.916544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:12.979295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 942/1900] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:12 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:12.983880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.047757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1622df84-d8b0-4a13-a2dc-5ac0a9da9793 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:13.052466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:13.053103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.053377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.053675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.054303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:13.056223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:13.086855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 959/1901] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:12 2022] POST /identity/v3/projects => generated 387 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:13.088220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.099319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b555a0-a0f3-4442-b3a9-8230661b4d0c None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:13.130156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:13.182813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:13.183734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.184197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.184680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.185548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:13.188384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:13.220975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 943/1902] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:13 2022] POST /identity/v3/users => generated 300 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:13.221768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.233733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76426b-4daf-4dd6-b09a-a12a4e411165 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:13.254491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:13.255537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.255889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.256373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.256788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:13.258669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:13.275811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 960/1903] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:13 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:13.279536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.289898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23adfd75-bbef-498f-a27b-129a397514fb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:13.294954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:13.295761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.296119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.296467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] PATH_INFO: `/v3/projects/e3ca204f74e641bca573b96cb01c3221/users/824548eaeb7b4390acc202b15b118424/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.327994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3ca204f74e641bca573b96cb01c3221, user_id=824548eaeb7b4390acc202b15b118424, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:13.330418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:13.354445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 944/1904] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:13 2022] PUT /identity/v3/projects/e3ca204f74e641bca573b96cb01c3221/users/824548eaeb7b4390acc202b15b118424/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:13.358262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.377639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2724ac71-d042-44e1-8631-42c14d315b33 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:13.523738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:13.542368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307a3bbb-5d57-43da-b569-2f5968e41e1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:13.543373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307a3bbb-5d57-43da-b569-2f5968e41e1c admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.544315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307a3bbb-5d57-43da-b569-2f5968e41e1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.544806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307a3bbb-5d57-43da-b569-2f5968e41e1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.548906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307a3bbb-5d57-43da-b569-2f5968e41e1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:13.558049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307a3bbb-5d57-43da-b569-2f5968e41e1c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:13.602167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 961/1905] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:13 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:13.606030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.638610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8cc252-3172-400b-ade8-7ed3e9be11c1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:13.647576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:13.649264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.649686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.650149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] PATH_INFO: `/v3/projects/e3ca204f74e641bca573b96cb01c3221/users/824548eaeb7b4390acc202b15b118424/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.655665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3ca204f74e641bca573b96cb01c3221, user_id=824548eaeb7b4390acc202b15b118424, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:13.661518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:13.680007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 945/1906] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:13 2022] PUT /identity/v3/projects/e3ca204f74e641bca573b96cb01c3221/users/824548eaeb7b4390acc202b15b118424/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:13.692135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:13.692888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:13.693252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:13.753152 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] MFA Rules not processed for user `824548eaeb7b4390acc202b15b118424`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:13.755485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:13.908549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 962/1907] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:13 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:13.977073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:13.979798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:14.057205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef2ceed-e5b9-4c6c-9244-216a3ba93de0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:14.060832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e91532f-6f58-4c78-95db-6a21ee1f978c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:14.061468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91532f-6f58-4c78-95db-6a21ee1f978c service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:14.061747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91532f-6f58-4c78-95db-6a21ee1f978c service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:14.062025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91532f-6f58-4c78-95db-6a21ee1f978c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:14.071278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e91532f-6f58-4c78-95db-6a21ee1f978c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:14.072703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e91532f-6f58-4c78-95db-6a21ee1f978c service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:14.082630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 946/1908] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:13 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:14.890181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:14.952833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952135cc-42a7-4ba9-b2f3-b31dd30e1dee None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:14.958330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:14.959441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:14.959441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:14.959614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] PATH_INFO: `/v3/users/51b9642eccfa4f7bbf5362489763a2af` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:14.960617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51b9642eccfa4f7bbf5362489763a2af)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:14.963532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:15.019242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51b9642eccfa4f7bbf5362489763a2af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.031499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51b9642eccfa4f7bbf5362489763a2af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.039178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51b9642eccfa4f7bbf5362489763a2af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.039905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51b9642eccfa4f7bbf5362489763a2af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.040528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51b9642eccfa4f7bbf5362489763a2af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.049200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:15.055989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 963/1909] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:14 2022] DELETE /identity/v3/users/51b9642eccfa4f7bbf5362489763a2af => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:15.060963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:15.061944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:15.062274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:15.082957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:15.084186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:15.143627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 947/1910] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:15.153335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:15.154802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:15.210663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d31096-7ecd-44c1-b831-93cb8a817c94 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:15.214232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:15.214832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:15.215115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:15.215388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:15.224682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:15.226143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:15.235594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 964/1911] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:15.505198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:15.506710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:15.521766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0560514-6fbc-4b17-94cf-61d09eecdac2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:15.527592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:15.528430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:15.528819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:15.529238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] PATH_INFO: `/v3/projects/57f2d6b240a847e7aa1c748a6464005b` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:15.530384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57f2d6b240a847e7aa1c748a6464005b)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:15.532290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:15.544594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:15.584515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57f2d6b240a847e7aa1c748a6464005b', 'request_id': 'req-5d3abdea-25bb-497f-9ed5-20d27ed4a414'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.592071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57f2d6b240a847e7aa1c748a6464005b', 'request_id': 'req-5d3abdea-25bb-497f-9ed5-20d27ed4a414'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.599554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57f2d6b240a847e7aa1c748a6464005b', 'request_id': 'req-5d3abdea-25bb-497f-9ed5-20d27ed4a414'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.610614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] The token cache is being invalidate because project 57f2d6b240a847e7aa1c748a6464005b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:15.610902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.611558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.613145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 948/1912] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:15 2022] DELETE /identity/v3/projects/57f2d6b240a847e7aa1c748a6464005b => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:15.617079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:15.626735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c30285-840a-4df5-853e-0f9c7c9f4b64 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:15.681591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:15.682239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:15.682519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:15.682794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] PATH_INFO: `/v3/users/824548eaeb7b4390acc202b15b118424` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:15.683913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=824548eaeb7b4390acc202b15b118424)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:15.685878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:15.732427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '824548eaeb7b4390acc202b15b118424'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.744104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '824548eaeb7b4390acc202b15b118424'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.750900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '824548eaeb7b4390acc202b15b118424'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.751559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '824548eaeb7b4390acc202b15b118424'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.752166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-db6b1bdd-877d-4f23-ac4d-c464ed7610d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '824548eaeb7b4390acc202b15b118424'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:15.766260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 965/1913] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:15 2022] DELETE /identity/v3/users/824548eaeb7b4390acc202b15b118424 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:16.072614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.074108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:16.090446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3abdea-25bb-497f-9ed5-20d27ed4a414 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.160251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.161112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.161413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.161801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] PATH_INFO: `/v3/projects/e3ca204f74e641bca573b96cb01c3221` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.163244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3ca204f74e641bca573b96cb01c3221)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.165256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.217907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3ca204f74e641bca573b96cb01c3221', 'request_id': 'req-8fce1a25-4e02-48e8-8ccf-63d4c4147992'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:16.225597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3ca204f74e641bca573b96cb01c3221', 'request_id': 'req-8fce1a25-4e02-48e8-8ccf-63d4c4147992'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:16.232159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3ca204f74e641bca573b96cb01c3221', 'request_id': 'req-8fce1a25-4e02-48e8-8ccf-63d4c4147992'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:16.245539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] The token cache is being invalidate because project e3ca204f74e641bca573b96cb01c3221 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:16.246208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:16.247104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8fce1a25-4e02-48e8-8ccf-63d4c4147992 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:16.248588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 949/1914] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:16 2022] DELETE /identity/v3/projects/e3ca204f74e641bca573b96cb01c3221 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:16.256669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.257163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.257346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.282644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:16.284323 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:16.298113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:16.350295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 966/1915] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:16.359380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.359681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.359951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.382245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:16.383490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:16.391799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 950/1916] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:16.396283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.407447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7656fbc-39c4-490a-b787-3ae530afd532 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.412920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.413554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.413871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.414209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.414930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:16.415254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.416914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.429491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 967/1917] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:16.433643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.443094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a7d6ed-0d02-48bd-ae8c-f33fc3b67d27 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.448654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.449325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.449623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.449894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.450510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.452372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.474515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:16.485044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 951/1918] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:16 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:16.486270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.496147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b2753-c51a-4421-b780-504f0b3b0c6a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.550421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.551013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.551289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.551560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.552185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.554172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.575709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 968/1919] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:16 2022] POST /identity/v3/users => generated 293 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:16.579852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.590685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a6c510-4a28-4cea-8bcc-83b22252ee2c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.596754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.597385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.597663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.599074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.599074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.600195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.611784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 952/1920] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:16.616475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.626460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58703db-5983-490b-886e-72d71a00511e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.632047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04ad4c3-d44f-4412-a0a9-a4c35d3870d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.632660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04ad4c3-d44f-4412-a0a9-a4c35d3870d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.632964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04ad4c3-d44f-4412-a0a9-a4c35d3870d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.633236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04ad4c3-d44f-4412-a0a9-a4c35d3870d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.633985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04ad4c3-d44f-4412-a0a9-a4c35d3870d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.635583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04ad4c3-d44f-4412-a0a9-a4c35d3870d8 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.646963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 969/1921] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:16.651441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.662082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c162c1e-03c3-4add-9ab6-6264b13549f3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.667360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.668013 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.668294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.668572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] PATH_INFO: `/v3/projects/8e6d42c16d82432f8688533d86bec993/users/435bd58ca81d49958432c47c99adc19c/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.683099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e6d42c16d82432f8688533d86bec993, user_id=435bd58ca81d49958432c47c99adc19c, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.685818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.700030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 953/1922] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:16 2022] PUT /identity/v3/projects/8e6d42c16d82432f8688533d86bec993/users/435bd58ca81d49958432c47c99adc19c/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:16.711733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.712050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.712330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.740625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] MFA Rules not processed for user `435bd58ca81d49958432c47c99adc19c`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:16.741878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:16.804424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 970/1923] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:16.816166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.818081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:16.877072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd65f2-138a-4758-9e4c-dd88c77b32c3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:16.880947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61897c3f-089e-40a0-a6db-6637a75adcb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:16.881566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61897c3f-089e-40a0-a6db-6637a75adcb5 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:16.881847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61897c3f-089e-40a0-a6db-6637a75adcb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:16.882142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61897c3f-089e-40a0-a6db-6637a75adcb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:16.890905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61897c3f-089e-40a0-a6db-6637a75adcb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:16.892336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61897c3f-089e-40a0-a6db-6637a75adcb5 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:16.902038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 954/1924] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:16 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:16.950697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:16.976699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:17.056001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbb4565-8e45-44cc-b2ac-21512513f015 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:17.061671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:17.062522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:17.062896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:17.063299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] PATH_INFO: `/v3/users/435bd58ca81d49958432c47c99adc19c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:17.065178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=435bd58ca81d49958432c47c99adc19c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:17.067598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:17.131369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '435bd58ca81d49958432c47c99adc19c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.158835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '435bd58ca81d49958432c47c99adc19c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.169224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '435bd58ca81d49958432c47c99adc19c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.170197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '435bd58ca81d49958432c47c99adc19c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.170918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '435bd58ca81d49958432c47c99adc19c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.189237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 971/1925] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:16 2022] DELETE /identity/v3/users/435bd58ca81d49958432c47c99adc19c => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:17.195348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:17.195877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:17.196003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:17.222135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:17.223552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:17.294049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 955/1926] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:17.307718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:17.309240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:17.376884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b53ab2-bbd9-426d-8329-685b780f2b74 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:17.381227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:17.381369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:17.381598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:17.381868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:17.394540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:17.396400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:17.423508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 972/1927] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:17.740793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:17.742994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:17.765764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df551901-2918-4cb8-990f-86bfadc8330e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:17.778065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:17.778894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:17.779357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:17.779637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] PATH_INFO: `/v3/users/408ae9f6dd9245c4b1e707e0c5bc44df` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:17.781327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] RBAC: Authorizing `identity:delete_user(user_id=408ae9f6dd9245c4b1e707e0c5bc44df)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:17.784614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:17.869324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '408ae9f6dd9245c4b1e707e0c5bc44df'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.876847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '408ae9f6dd9245c4b1e707e0c5bc44df'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.877640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '408ae9f6dd9245c4b1e707e0c5bc44df'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.879011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '408ae9f6dd9245c4b1e707e0c5bc44df'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.886710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '408ae9f6dd9245c4b1e707e0c5bc44df'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:17.888174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:17.909017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:17.915028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:17.928670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 956/1928] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:17 2022] DELETE /identity/v3/users/408ae9f6dd9245c4b1e707e0c5bc44df => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:17.929615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:17.931226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:17.941908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0774be99-a1dc-4dce-8ab7-3087dac46859 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:17.985653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:18.008762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c699139e-bf0c-48ab-a3d3-a433c6348836 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.013003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.013718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.014146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.014418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] PATH_INFO: `/v3/projects/8e6d42c16d82432f8688533d86bec993` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.015597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e6d42c16d82432f8688533d86bec993)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.016370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.017132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.017563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.017693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.017850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] PATH_INFO: `/v3/users/2097a3ea6f7249bbbd496d49da9beb31` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.019067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] RBAC: Authorizing `identity:delete_user(user_id=2097a3ea6f7249bbbd496d49da9beb31)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.020992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.120183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2097a3ea6f7249bbbd496d49da9beb31'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.127512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e6d42c16d82432f8688533d86bec993', 'request_id': 'req-450b19ef-627d-4af5-ac3c-665b292350e9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.137673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2097a3ea6f7249bbbd496d49da9beb31'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.138522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2097a3ea6f7249bbbd496d49da9beb31'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.140284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e6d42c16d82432f8688533d86bec993', 'request_id': 'req-450b19ef-627d-4af5-ac3c-665b292350e9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.140558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2097a3ea6f7249bbbd496d49da9beb31'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.148080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e6d42c16d82432f8688533d86bec993', 'request_id': 'req-450b19ef-627d-4af5-ac3c-665b292350e9'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.152451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e7de4066-6a8b-480b-8ca2-dfbcfce2f7df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2097a3ea6f7249bbbd496d49da9beb31'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.162205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] The token cache is being invalidate because project 8e6d42c16d82432f8688533d86bec993 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:18.162662 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.163506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-450b19ef-627d-4af5-ac3c-665b292350e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:18.165359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 973/1929] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:17 2022] DELETE /identity/v3/projects/8e6d42c16d82432f8688533d86bec993 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:18.166998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ff0a8c-68ab-4e30-9275-ff2bde858eb1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.167380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ff0a8c-68ab-4e30-9275-ff2bde858eb1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.167722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ff0a8c-68ab-4e30-9275-ff2bde858eb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.174274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 957/1930] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:17 2022] DELETE /identity/v3/users/2097a3ea6f7249bbbd496d49da9beb31 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:18.175687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.176046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.176444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.188949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e1ff0a8c-68ab-4e30-9275-ff2bde858eb1 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:18.191298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e1ff0a8c-68ab-4e30-9275-ff2bde858eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:18.197720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:18.198915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:18.266675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 974/1931] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:18.270258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:18.272112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8816bc71-97ec-4fdc-be1a-e7a1529c9fa2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.272706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8816bc71-97ec-4fdc-be1a-e7a1529c9fa2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.273021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8816bc71-97ec-4fdc-be1a-e7a1529c9fa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.281152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 958/1932] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:18.282034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.290020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:18.301911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8816bc71-97ec-4fdc-be1a-e7a1529c9fa2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:18.303181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8816bc71-97ec-4fdc-be1a-e7a1529c9fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:18.311405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 975/1933] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:18.312645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.312958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.313234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.334332 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:18.335461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:18.344912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 976/1934] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:18.349757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.358005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da10085f-9f7e-4bdb-820f-1a656de31282 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.359735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434b4b78-64f9-43cf-be71-9e557a75e420 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.362183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.363090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.363678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.363957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.364078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.364794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.365104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.365379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.374463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.375829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.376174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.377695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.385779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 959/1935] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:18.386941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.388196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 977/1936] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:18.398493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26a94c1-445c-47f2-9b5c-6226a3cdfa85 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.403884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.404593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.404834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.405139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.405793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:18.406125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.407752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.434365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 960/1937] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:18.438584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.450272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadeffa5-9c37-4d10-8f58-e65050ae745c service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.456599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.457807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.458091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.458381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.459044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.461289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.505068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 978/1938] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/projects => generated 359 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:18.508720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.520139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8aae29-b0be-46fa-8656-1771f45e5345 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.590938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.591487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.591907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.592691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.593142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.595352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.634002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 961/1939] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/users => generated 280 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:18.634304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.651427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb336580-b2b0-4f0b-9047-15c69b5cf425 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.661962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.662713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.663049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.663394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.664256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.666631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.693479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 979/1940] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:18 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:18.702739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.737842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9d9c18-fe3f-4b4a-a14f-b0b072e1fd98 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.743725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3189d2b9-53bb-45f9-8bff-7ac5bef6d3f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.744501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3189d2b9-53bb-45f9-8bff-7ac5bef6d3f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.745005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3189d2b9-53bb-45f9-8bff-7ac5bef6d3f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.745386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3189d2b9-53bb-45f9-8bff-7ac5bef6d3f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.746788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3189d2b9-53bb-45f9-8bff-7ac5bef6d3f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.749069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3189d2b9-53bb-45f9-8bff-7ac5bef6d3f4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.763854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 962/1941] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:18 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:18.770583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:18.784688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91faf37-4747-426f-bb58-a4ec113b9fb6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:18.793633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:18.794822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.794822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.795049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] PATH_INFO: `/v3/projects/51ea603b1b8d435c987baa899eebeb32/users/2c526207d0a34dbfb142ec95ce07731d/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.801706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:18.821859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=51ea603b1b8d435c987baa899eebeb32, user_id=2c526207d0a34dbfb142ec95ce07731d, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:18.863962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:18.913040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 980/1942] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:18 2022] PUT /identity/v3/projects/51ea603b1b8d435c987baa899eebeb32/users/2c526207d0a34dbfb142ec95ce07731d/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:18.928365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:18.928365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:18.928365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:18.995209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] MFA Rules not processed for user `2c526207d0a34dbfb142ec95ce07731d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:18.996648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.119472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:19.121879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.135930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:19.148952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 963/1943] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:18 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:19.182859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:19.184310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.268098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f803e34-fdce-4ef8-9ffc-d5aedc0dc5ff admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:19.274517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:19.275521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.276046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.276382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] PATH_INFO: `/v3/projects/4c6dfa04d2f24ac68cfbf3f544d30797` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.277813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c6dfa04d2f24ac68cfbf3f544d30797)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:19.280168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:19.381010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ca03a-1915-45d1-ba73-7ab6e4721d53 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:19.398629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c6dfa04d2f24ac68cfbf3f544d30797', 'request_id': 'req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.412455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c6dfa04d2f24ac68cfbf3f544d30797', 'request_id': 'req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.428636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c6dfa04d2f24ac68cfbf3f544d30797', 'request_id': 'req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.447904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] The token cache is being invalidate because project 4c6dfa04d2f24ac68cfbf3f544d30797 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:19.448203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.450159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.456762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 981/1944] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:19 2022] DELETE /identity/v3/projects/4c6dfa04d2f24ac68cfbf3f544d30797 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:19.457672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:19.461172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.515055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:19.515769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.516125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.519392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] PATH_INFO: `/v3/projects/77947ca6b22e4f03b17317021a2bb7fa` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.522819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77947ca6b22e4f03b17317021a2bb7fa)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:19.525588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:19.612389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77947ca6b22e4f03b17317021a2bb7fa', 'request_id': 'req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.617705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb9c6b-f69f-4b98-8ab2-1a207b371afa admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:19.623022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77947ca6b22e4f03b17317021a2bb7fa', 'request_id': 'req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.633371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77947ca6b22e4f03b17317021a2bb7fa', 'request_id': 'req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.646478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] The token cache is being invalidate because project 77947ca6b22e4f03b17317021a2bb7fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:19.646850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.647745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4f9f49aa-ec7a-498a-9b2f-5aba3272a296 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.652491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 964/1945] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:19 2022] DELETE /identity/v3/projects/77947ca6b22e4f03b17317021a2bb7fa => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:19.652491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.652491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.652491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.674618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:19.676133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.685395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:19.730632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:19.731566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:19.732713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.732713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.735187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.738357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.794234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 965/1946] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:19.795836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:19.800660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.822472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cfc21c-b888-4254-9f5f-482186fa144a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:19.835855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:19.837309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.837959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.838459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] PATH_INFO: `/v3/users/0e72daa1f1584c47ad375cf8143643b1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.839924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:19.840367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e72daa1f1584c47ad375cf8143643b1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:19.842124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686f703c-b371-42a4-9072-05ecc044b02c service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:19.842899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:19.857387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 982/1947] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:19 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:19.869441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.869903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.870997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.897688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:19.899236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.909524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e72daa1f1584c47ad375cf8143643b1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.910112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 983/1948] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:19.911131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:19.912693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:19.923425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e72daa1f1584c47ad375cf8143643b1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.923889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e72daa1f1584c47ad375cf8143643b1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.925121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e72daa1f1584c47ad375cf8143643b1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.933972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e72daa1f1584c47ad375cf8143643b1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:19.938400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7fc0d2-5bcf-426d-85f0-26343e4a8a55 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:19.943880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:19.944716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:19.945011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:19.945416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] PATH_INFO: `/v3/users/2c526207d0a34dbfb142ec95ce07731d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:19.946589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c526207d0a34dbfb142ec95ce07731d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:19.948770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:19.961462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 966/1949] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:19 2022] DELETE /identity/v3/users/0e72daa1f1584c47ad375cf8143643b1 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:19.962655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:19.964396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:20.005834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c526207d0a34dbfb142ec95ce07731d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:20.026553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c526207d0a34dbfb142ec95ce07731d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:20.036138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c526207d0a34dbfb142ec95ce07731d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:20.037371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c526207d0a34dbfb142ec95ce07731d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:20.037990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e59fbed4-6f8a-4022-8496-1882f08c06dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c526207d0a34dbfb142ec95ce07731d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:20.054918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c2887c-36d0-43f4-941e-1d5d7885d039 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.061000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 984/1950] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:19 2022] DELETE /identity/v3/users/2c526207d0a34dbfb142ec95ce07731d => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:20.061747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.062385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.062720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.062999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.063663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:20.063972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.065998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.067342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.068015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.068598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.083775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 967/1951] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:20.084708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.087325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:20.100228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:20.101465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d840abb9-ccd8-4664-b1a7-95fd10eb138f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.102703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:20.168334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:20.171446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:20.197470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 985/1952] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:20.200583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.201259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.201548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.201861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.202530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.204432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.229361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.231561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:20.265418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 968/1953] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:20 2022] POST /identity/v3/projects => generated 383 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:20.270875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.325930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70507139-8a06-4450-bb78-dd791d70d96b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.387578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.388745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.389073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.389475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.422221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fbba2d-607f-433e-bbe9-2182117e4d8d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.433617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.438152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.440126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.442305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.442305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.448246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.477976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 969/1954] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:20 2022] POST /identity/v3/users => generated 292 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:20.479129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.491432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.492261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c23f0d-a644-4d34-b36c-c81c354d0f89 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.493075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.498685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.499557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.500242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.500521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.501201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.503457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.503607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 986/1955] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:20.519364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 970/1956] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:20.520045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.543631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7ff305-e3d4-4e58-89e2-e20cc57a6160 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.551584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46cd727-3c88-43e8-9120-c557dc4d08ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.552172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46cd727-3c88-43e8-9120-c557dc4d08ac admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.552500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46cd727-3c88-43e8-9120-c557dc4d08ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.552782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46cd727-3c88-43e8-9120-c557dc4d08ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.554334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46cd727-3c88-43e8-9120-c557dc4d08ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.557333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46cd727-3c88-43e8-9120-c557dc4d08ac admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.588928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 987/1957] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:20.590328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.601851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb76a8a-c76f-474f-ab7b-d7844ef94541 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.607719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.608700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.609321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.609321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] PATH_INFO: `/v3/projects/5fbf9c9a34c54fcfac32c229d3660449/users/a800b175c9a9462089d86da9e226338c/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.633156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fbf9c9a34c54fcfac32c229d3660449, user_id=a800b175c9a9462089d86da9e226338c, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.636045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.657092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 971/1958] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:20 2022] PUT /identity/v3/projects/5fbf9c9a34c54fcfac32c229d3660449/users/a800b175c9a9462089d86da9e226338c/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:20.667913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.668109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.668354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.701947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] MFA Rules not processed for user `a800b175c9a9462089d86da9e226338c`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:20.703384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:20.742654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:20.787132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 988/1959] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:20.790127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.873217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01813c39-5b57-45d8-8138-b5f4dd7b1863 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.879531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:20.880363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:20.880932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:20.881379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:20.882210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:20.884456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:20.917445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 972/1960] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:20 2022] POST /identity/v3/projects => generated 385 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:20.918804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.931575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f43ab39-eade-42da-83f7-89584ebcad37 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:20.932682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:20.936415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:20.949806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:21.013336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.014249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.014658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.014947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.015594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.018203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.026974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0899c63-978e-4973-aa2d-ca1f7811acb0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.033043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.033720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.034458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.034748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] PATH_INFO: `/v3/projects/fb347b0fdfbe4276b34799d40a000c59` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.036135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb347b0fdfbe4276b34799d40a000c59)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.038256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.045954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 989/1961] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:20 2022] POST /identity/v3/users => generated 293 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:21.046791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.048498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.068336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4337fb97-8262-4839-812f-c0601f1f7d64 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.074712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.075355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.075642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.075930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] PATH_INFO: `/v3/projects/51ea603b1b8d435c987baa899eebeb32` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.077624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=51ea603b1b8d435c987baa899eebeb32)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.079882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.095186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb347b0fdfbe4276b34799d40a000c59', 'request_id': 'req-7d2fa035-f414-4ede-8fa5-83585d736628'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.104094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb347b0fdfbe4276b34799d40a000c59', 'request_id': 'req-7d2fa035-f414-4ede-8fa5-83585d736628'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.113356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb347b0fdfbe4276b34799d40a000c59', 'request_id': 'req-7d2fa035-f414-4ede-8fa5-83585d736628'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.126403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] The token cache is being invalidate because project fb347b0fdfbe4276b34799d40a000c59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:21.126617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.127376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.132793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 973/1962] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:20 2022] DELETE /identity/v3/projects/fb347b0fdfbe4276b34799d40a000c59 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:21.133633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.135025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.141491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51ea603b1b8d435c987baa899eebeb32', 'request_id': 'req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.152580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51ea603b1b8d435c987baa899eebeb32', 'request_id': 'req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.173884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51ea603b1b8d435c987baa899eebeb32', 'request_id': 'req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.198511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] The token cache is being invalidate because project 51ea603b1b8d435c987baa899eebeb32 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:21.198902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.199750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5c582b6-2c37-4ae8-afec-5ff9b5061b9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:21.202781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 990/1963] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:21 2022] DELETE /identity/v3/projects/51ea603b1b8d435c987baa899eebeb32 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:21.204236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed2e0c-23d0-4e57-a3d4-77811cd7d7c1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.204579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed2e0c-23d0-4e57-a3d4-77811cd7d7c1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.204956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed2e0c-23d0-4e57-a3d4-77811cd7d7c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.219987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2fa035-f414-4ede-8fa5-83585d736628 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.229118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3bed2e0c-23d0-4e57-a3d4-77811cd7d7c1 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:21.230679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3bed2e0c-23d0-4e57-a3d4-77811cd7d7c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.282210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:21.296021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:21.303141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddad39e-04e8-4171-9022-3d3d3cc3e6d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.304188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddad39e-04e8-4171-9022-3d3d3cc3e6d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.304622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddad39e-04e8-4171-9022-3d3d3cc3e6d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.305211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddad39e-04e8-4171-9022-3d3d3cc3e6d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.306248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddad39e-04e8-4171-9022-3d3d3cc3e6d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.308799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddad39e-04e8-4171-9022-3d3d3cc3e6d2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.321644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 991/1964] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:21.323420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.323886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.324333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.332581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 974/1965] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:21.335755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0f7ed-6a90-4bec-8544-0747f0b9029e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.336962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0f7ed-6a90-4bec-8544-0747f0b9029e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.337465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de0f7ed-6a90-4bec-8544-0747f0b9029e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.350240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:21.352704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.360441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5de0f7ed-6a90-4bec-8544-0747f0b9029e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:21.362589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5de0f7ed-6a90-4bec-8544-0747f0b9029e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.364730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 992/1966] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:21.365825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.367684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.378144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 975/1967] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:21.379447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.379740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.380020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.383388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e68ca1-1b8d-47d7-8891-ff1d14746c3e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.388329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.388976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.390059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.390059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.390315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.391982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.401150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:21.402376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.406827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 993/1968] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:21.406827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.415377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 976/1969] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:21.416355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.419374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445dff7-577d-471c-9506-0f284ad67c0e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.425710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.426325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.427132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.427243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e761d5-6f8c-40fb-a5e2-85ef6c788e6a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.427681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.427681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:21.427907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.429584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.433360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.433652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.433927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.434581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] PATH_INFO: `/v3/projects/64209f68dacc47139f3eed0ab0d00205/users/46623dbd007b4f82998d3919b7e9119e/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.447362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 994/1970] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:21.447362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.454362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64209f68dacc47139f3eed0ab0d00205, user_id=46623dbd007b4f82998d3919b7e9119e, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.458257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.458548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d2de4d-1a52-4e95-ad79-d560d93aef21 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.464214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.464952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.465331 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.465583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.466301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:21.466667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.468636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891df280-4ae3-402b-b51b-0584e1fe6e7f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.484675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 977/1971] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:21 2022] PUT /identity/v3/projects/64209f68dacc47139f3eed0ab0d00205/users/46623dbd007b4f82998d3919b7e9119e/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:21.484675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.484972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 995/1972] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:21.490769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8040991d-e154-4714-980c-acb488267288 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.491060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8040991d-e154-4714-980c-acb488267288 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.491685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8040991d-e154-4714-980c-acb488267288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.536251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8040991d-e154-4714-980c-acb488267288 None None] MFA Rules not processed for user `46623dbd007b4f82998d3919b7e9119e`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:21.537947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8040991d-e154-4714-980c-acb488267288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:21.589067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ca9eb3-ea8b-4001-aad9-10cef59aa502 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.594743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.595360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.595640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.595987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.597023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.599920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.632968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 978/1973] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/projects => generated 381 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:21.634220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.635949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 996/1974] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:21.635949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8040991d-e154-4714-980c-acb488267288 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.648247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8040991d-e154-4714-980c-acb488267288 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.709897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3217c4-98d7-4bf6-bd56-3aa8571e9f3a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.710144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.710703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.711048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.711399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.712454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.714192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.714900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.715666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.716094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.716533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.717336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.719595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.739388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 997/1975] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/users => generated 291 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:21.740324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.751263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb41f1-cc2b-44e2-a055-5491d9f50c36 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.757113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.757748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 979/1976] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/projects => generated 383 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:21.758634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.761440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.761599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.761889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.762988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.764957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.778462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:21.806427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 998/1977] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:21.806427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.814064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:21.856847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5770a94-59aa-453d-a83c-7ffb22e1c565 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.899472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa56246-c678-4f25-a964-6e034481f408 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.906593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.907423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.907946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.909425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.910172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.913060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.933076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.934273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.934840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.935319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.936282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.938137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.943854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 999/1978] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/users => generated 292 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:21.944305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.957152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 980/1979] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:21.958725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:21.962038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a714c118-4542-4dcf-b7cc-52bd8313e392 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.970354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a283d71e-9667-4975-858d-c27df568b427 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:21.974868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.975497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.975825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.976104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.976737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:21.978924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:21.997283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:21.997979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:21.998549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:21.998921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:21.999643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.001352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.004841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1000/1980] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:21 2022] POST /identity/v3/users => generated 299 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:22.005546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.015579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 981/1981] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.016268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.016992 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a290d64a-150f-490d-8bd5-4ef3fd90bb98 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.023326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.023945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.024221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.024828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.025644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.026831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.030158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda08875-5e97-4482-9a39-9609806d98a5 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.035494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.036090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.036369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.036636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.037397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.039029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.042815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1001/1982] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.043796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.050365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 982/1983] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.051014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.062106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef607505-1a18-414d-b38b-edcfde1e5cd4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.062929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b600ab9b-a2e9-4f6c-a313-a3953ed94a2d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.067452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.067795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.068224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.068433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.068582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.068717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.069007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] PATH_INFO: `/v3/projects/8690cde79c8844b1a57f318293d0f5f5/users/f70dbcf39cdd49178867705333c094f7/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.069102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.069815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.071689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.084592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1002/1984] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.084889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8690cde79c8844b1a57f318293d0f5f5, user_id=f70dbcf39cdd49178867705333c094f7, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.087593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.088414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.099658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b512794-072b-4758-bd11-26e385f380b6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.110294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 983/1985] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:22 2022] PUT /identity/v3/projects/8690cde79c8844b1a57f318293d0f5f5/users/f70dbcf39cdd49178867705333c094f7/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:22.110294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.178771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02b5e88-c7be-498d-869c-37b529185c45 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.180950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ae5e6c-c4b1-407e-82b1-e4c55726649d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.181933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae5e6c-c4b1-407e-82b1-e4c55726649d admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.182401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae5e6c-c4b1-407e-82b1-e4c55726649d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.182685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae5e6c-c4b1-407e-82b1-e4c55726649d admin admin] PATH_INFO: `/v3/projects/754d0742bf9648e39239b72251c60fda/users/c73fe6208b544b8cbc7a2b2f8c9b0f5d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.184009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.184646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.184950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.185229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] PATH_INFO: `/v3/projects/968d4393b70f4260aea95f8860ed24cc/users/dea35bcb0d124a2ea034db0d5dd02393/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.199385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ae5e6c-c4b1-407e-82b1-e4c55726649d admin admin] RBAC: Authorizing `identity:create_grant(project_id=754d0742bf9648e39239b72251c60fda, user_id=c73fe6208b544b8cbc7a2b2f8c9b0f5d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.200591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=968d4393b70f4260aea95f8860ed24cc, user_id=dea35bcb0d124a2ea034db0d5dd02393, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.202106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ae5e6c-c4b1-407e-82b1-e4c55726649d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.203244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.205474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:22.216674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1003/1986] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:22 2022] PUT /identity/v3/projects/754d0742bf9648e39239b72251c60fda/users/c73fe6208b544b8cbc7a2b2f8c9b0f5d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:22.217950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.218285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.218671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.220638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 984/1987] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:22 2022] PUT /identity/v3/projects/968d4393b70f4260aea95f8860ed24cc/users/dea35bcb0d124a2ea034db0d5dd02393/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:22.225756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.252900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] MFA Rules not processed for user `f70dbcf39cdd49178867705333c094f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:22.254123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:22.282645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:22.306030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e72e04f-5384-4cac-8eb2-bb447c9650ef admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.311971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c00a201-4b74-4e78-adbe-4a08098d5088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.312585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c00a201-4b74-4e78-adbe-4a08098d5088 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.312860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c00a201-4b74-4e78-adbe-4a08098d5088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.313295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c00a201-4b74-4e78-adbe-4a08098d5088 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.313945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c00a201-4b74-4e78-adbe-4a08098d5088 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.316368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c00a201-4b74-4e78-adbe-4a08098d5088 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.328078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 985/1988] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.329693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4513f076-fea8-4ffc-aa91-5177cd746d12 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.329977 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4513f076-fea8-4ffc-aa91-5177cd746d12 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.330248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4513f076-fea8-4ffc-aa91-5177cd746d12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.334249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1004/1989] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:22.338240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.349506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5618d3c6-e79e-4491-a6a4-1d2fc0c481dc None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.356471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.357192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.358033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.358268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] PATH_INFO: `/v3/projects/754d0742bf9648e39239b72251c60fda/users/c73fe6208b544b8cbc7a2b2f8c9b0f5d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.360525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] RBAC: Authorizing `identity:create_grant(project_id=754d0742bf9648e39239b72251c60fda, user_id=c73fe6208b544b8cbc7a2b2f8c9b0f5d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.361792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4513f076-fea8-4ffc-aa91-5177cd746d12 None None] MFA Rules not processed for user `dea35bcb0d124a2ea034db0d5dd02393`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:22.363866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4513f076-fea8-4ffc-aa91-5177cd746d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:22.363997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.376658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1005/1990] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:22 2022] PUT /identity/v3/projects/754d0742bf9648e39239b72251c60fda/users/c73fe6208b544b8cbc7a2b2f8c9b0f5d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:22.377580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.458821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 986/1991] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:22.459431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.459681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.459959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.465683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556e4eb6-e847-48ed-b5e9-79fce5b58332 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.470867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.472959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.472959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.472959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.473489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.476040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.491286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] MFA Rules not processed for user `c73fe6208b544b8cbc7a2b2f8c9b0f5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:22.493334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:22.511606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1006/1992] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/projects => generated 381 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:22.511606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.602663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 987/1993] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:22.603643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113e5c39-2d47-41f9-ac9b-ebfd93431c1a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.604205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.608580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.609224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.609696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.609913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.610388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.611364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:22.612437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.619901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404ea898-8861-4ab5-8317-1a61655df20d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.625439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.626062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.626342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.626612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.627444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.630044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.651461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1007/1994] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/projects => generated 385 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:22.651461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.652736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:22.660763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 988/1995] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/users => generated 297 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:22.661954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.673564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f46298-ef99-4048-9f0d-22a16a5bdcde admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.683537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:22.717879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f959fb-7e1e-4e7a-8890-82e82bd0b310 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.722076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.722684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.722962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.723236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.725068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:22.744837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.745673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.746024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.746360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.747069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.749130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.776807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 989/1996] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:22 2022] POST /identity/v3/users => generated 293 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:22.781609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.792146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d538c5-4d72-4218-8de8-424f787aaccb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.795450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.797131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.797909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.798676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.798960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.799228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.800504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.802156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.809297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1008/1997] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:22.810227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.816901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 990/1998] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.818535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.828671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c943cfd9-04b0-46a2-8aa1-b8630ad4bc2d service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.829254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70474fcc-9eb1-4d57-b97c-93fc267639cf admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.834482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.835156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.835534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.835842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.836080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.836741 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.836860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.837062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.837440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] PATH_INFO: `/v3/projects/4482eb58d6fe46eb970a64dc2f834e3a/users/d7d10dcf130643b780bac8d257d39fe1/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.838579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.860652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4482eb58d6fe46eb970a64dc2f834e3a, user_id=d7d10dcf130643b780bac8d257d39fe1, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.864520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.869638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1009/1999] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.873950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.888924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 991/2000] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:22 2022] PUT /identity/v3/projects/4482eb58d6fe46eb970a64dc2f834e3a/users/d7d10dcf130643b780bac8d257d39fe1/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:22.889720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a5e017-a4d0-4fbc-a4ae-fb68a4a315f4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.892447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.967717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc84d5c-4f06-472a-aa9c-35c031ce359c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:22.968710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.969283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.969561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.969955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.970594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.972743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.975060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:22.976111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:22.976539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:22.977480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:22.978273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:22.980402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:22.984725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1010/2001] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:22.989347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:22.997881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 992/2002] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:23.007167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb0b71-d039-4345-a154-f91da5d5b201 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.016896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.018195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2328b8c-3a74-44fd-9996-3f1a121298ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.020985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2328b8c-3a74-44fd-9996-3f1a121298ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.021278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2328b8c-3a74-44fd-9996-3f1a121298ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.021612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2328b8c-3a74-44fd-9996-3f1a121298ad admin admin] PATH_INFO: `/v3/projects/729aec6772cb4365aa3c801ccf482a5c/users/54da39f14bd34e6996e6ae4067c39300/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.043674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2328b8c-3a74-44fd-9996-3f1a121298ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=729aec6772cb4365aa3c801ccf482a5c, user_id=54da39f14bd34e6996e6ae4067c39300, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.045116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c4e8ca-412e-4ceb-8f86-7125d14d0f27 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.046905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2328b8c-3a74-44fd-9996-3f1a121298ad admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.055737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec10032-f8bd-4848-8318-7140233be855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.056676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec10032-f8bd-4848-8318-7140233be855 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.057266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec10032-f8bd-4848-8318-7140233be855 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.057702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec10032-f8bd-4848-8318-7140233be855 admin admin] PATH_INFO: `/v3/projects/4482eb58d6fe46eb970a64dc2f834e3a/users/d7d10dcf130643b780bac8d257d39fe1/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.062485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec10032-f8bd-4848-8318-7140233be855 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4482eb58d6fe46eb970a64dc2f834e3a, user_id=d7d10dcf130643b780bac8d257d39fe1, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.065649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec10032-f8bd-4848-8318-7140233be855 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.069271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1011/2003] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:22 2022] PUT /identity/v3/projects/729aec6772cb4365aa3c801ccf482a5c/users/54da39f14bd34e6996e6ae4067c39300/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:23.086681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.086839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.087231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.094800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 993/2004] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:23 2022] PUT /identity/v3/projects/4482eb58d6fe46eb970a64dc2f834e3a/users/d7d10dcf130643b780bac8d257d39fe1/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:23.105441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.105881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.106220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.132969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] MFA Rules not processed for user `54da39f14bd34e6996e6ae4067c39300`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:23.134370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:23.143021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] MFA Rules not processed for user `d7d10dcf130643b780bac8d257d39fe1`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:23.144820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:23.159856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:23.169826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:23.241379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1012/2005] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:23.244718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.288910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 994/2006] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:23 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:23.300901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.330068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f83f730-693f-447f-b9d4-78d94e65a5d6 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.335349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.335963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.336352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.336558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.337370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.339454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.362086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc6f6c7-1c34-4148-bc0c-2fa5f69d6cda None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.365971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.366589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.366864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1013/2007] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:23 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:23.368979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.369346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.371048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.379677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.381339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.391066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 995/2008] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:23 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:23.442071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c485a42-06ed-4a8c-80e5-c72e6f2661b4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.449887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.450562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.450876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.451308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.451986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.454459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.476786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.479813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1014/2009] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:23 2022] POST /identity/v3/users => generated 299 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:23.483739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.486804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592b6345-89f1-4601-a792-6ccee1309633 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.495829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd310d-2a32-41db-bf29-94e8afb4869f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.500455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.501104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.501386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.501658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.502303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.503988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.514848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1015/2010] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:23.519618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.530301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b181ac5b-ee57-4f90-890a-fd1a1a5d216b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.535526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.536178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.536455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.536727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] PATH_INFO: `/v3/projects/9694c99c58ee4d60b40edb6e9fc93102/users/5dcc9ed5050144bd81a6841728f8b13c/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.548003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.548624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.548990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.549270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.554586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9694c99c58ee4d60b40edb6e9fc93102, user_id=5dcc9ed5050144bd81a6841728f8b13c, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.557914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.586913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1016/2011] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:23 2022] PUT /identity/v3/projects/9694c99c58ee4d60b40edb6e9fc93102/users/5dcc9ed5050144bd81a6841728f8b13c/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:23.592028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.608952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:23.649628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.651068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.665069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 996/2012] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:23 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:23.687408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd56762-81ff-4b1b-b83b-54831b504e57 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.692683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013bc99-f0c0-4678-87bb-4381f6b68e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.693411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013bc99-f0c0-4678-87bb-4381f6b68e87 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.693876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013bc99-f0c0-4678-87bb-4381f6b68e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.694087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013bc99-f0c0-4678-87bb-4381f6b68e87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.694773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013bc99-f0c0-4678-87bb-4381f6b68e87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.696305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013bc99-f0c0-4678-87bb-4381f6b68e87 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.707454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1017/2013] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:23 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:23.711470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.723471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc3bfc-f7d9-4eb2-a529-351cfd96643e service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.728785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.729554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.729808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.730209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] PATH_INFO: `/v3/projects/9694c99c58ee4d60b40edb6e9fc93102/users/5dcc9ed5050144bd81a6841728f8b13c/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.732145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9694c99c58ee4d60b40edb6e9fc93102, user_id=5dcc9ed5050144bd81a6841728f8b13c, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:23.735910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:23.738681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:23.751300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 997/2014] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:23 2022] PUT /identity/v3/projects/9694c99c58ee4d60b40edb6e9fc93102/users/5dcc9ed5050144bd81a6841728f8b13c/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:23.761501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.761782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.762053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:23.797847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] MFA Rules not processed for user `5dcc9ed5050144bd81a6841728f8b13c`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:23.799176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:23.874593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1018/2015] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:23.886938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:23.949759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7668ab-09b7-446d-8d80-09c2bf24beeb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:23.953489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:23.954090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:23.954370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:23.954640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:24.015268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:24.016681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:24.026254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 998/2016] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:24.527102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:24.528491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:24.540092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0ff60-fc9c-4637-85c5-d8ac94fdb2a6 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:24.543701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:24.544301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:24.544580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:24.544856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:24.604805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:24.606298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:24.615828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1019/2017] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:24 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:25.490664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:25.501210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b19bc-af87-483a-bd8d-2c1a19b4f05e service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:25.510846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:25.511980 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:25.512542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:25.513359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:25.604501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:25.606028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:25.618507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 999/2018] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:25 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:25.783142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:25.794475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d03dc7f-0ce8-4064-acd9-85dc6c349a1d service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:25.800413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:25.801188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:25.801566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:25.801947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:25.831654 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:25.869473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:25.871150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:25.885619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1020/2019] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:25 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:26.736630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:26.747011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7c887c-e416-4b5d-b56b-0cf17e544d16 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:26.751206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fc1cf3-6818-4fe6-8a78-b7af65b7f6cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:26.751468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fc1cf3-6818-4fe6-8a78-b7af65b7f6cd service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:26.751748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fc1cf3-6818-4fe6-8a78-b7af65b7f6cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:26.752022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fc1cf3-6818-4fe6-8a78-b7af65b7f6cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:26.765110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fc1cf3-6818-4fe6-8a78-b7af65b7f6cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:26.766681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fc1cf3-6818-4fe6-8a78-b7af65b7f6cd service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:26.779050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1000/2020] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:26 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:27.747216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:27.748661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:27.816838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6235f-3167-4e3c-8a8f-dffa4e3a0e52 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:27.823832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:27.824649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:27.824956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:27.825229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] PATH_INFO: `/v3/users/31a10e99dbdc493da395f9a1cf63270f` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:27.826363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31a10e99dbdc493da395f9a1cf63270f)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:27.828292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:27.875626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31a10e99dbdc493da395f9a1cf63270f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:27.888127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31a10e99dbdc493da395f9a1cf63270f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:27.895551 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31a10e99dbdc493da395f9a1cf63270f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:27.896229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31a10e99dbdc493da395f9a1cf63270f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:27.896843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31a10e99dbdc493da395f9a1cf63270f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:27.913126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1021/2021] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:27 2022] DELETE /identity/v3/users/31a10e99dbdc493da395f9a1cf63270f => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:27.914799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:27.915199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:27.915386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:27.934632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:27.935985 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:27.962031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:28.008312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1001/2022] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:28.017774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:28.019698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:28.067355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:28.082416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4276b25-32d2-4b27-92a3-62c01af2de41 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:28.086102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:28.086709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:28.086987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:28.087267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:28.096087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:28.097585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:28.106864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1022/2023] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:28.377795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:28.379316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:28.393790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42643252-8c7f-42ec-ace6-c6e0fcb018a9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:28.399504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:28.400206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:28.400390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:28.400665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] PATH_INFO: `/v3/projects/51872300f3994c7d859ce74a47015d09` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:28.401841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51872300f3994c7d859ce74a47015d09)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:28.403744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:28.450560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51872300f3994c7d859ce74a47015d09', 'request_id': 'req-192cba3f-6832-49d9-ae17-81ca731c72c2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.458227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51872300f3994c7d859ce74a47015d09', 'request_id': 'req-192cba3f-6832-49d9-ae17-81ca731c72c2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.464885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51872300f3994c7d859ce74a47015d09', 'request_id': 'req-192cba3f-6832-49d9-ae17-81ca731c72c2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.475375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] The token cache is being invalidate because project 51872300f3994c7d859ce74a47015d09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:28.475676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.476326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.477870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1002/2024] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:28 2022] DELETE /identity/v3/projects/51872300f3994c7d859ce74a47015d09 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:28.481818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:28.491591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537404c-ef2e-4df8-b33b-5e0f57db9a97 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:28.546326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:28.546954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:28.547238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:28.547513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] PATH_INFO: `/v3/users/1aef935d95ac40ca82cddf1a709ff671` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:28.548649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1aef935d95ac40ca82cddf1a709ff671)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:28.551400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:28.597161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1aef935d95ac40ca82cddf1a709ff671'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.609041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1aef935d95ac40ca82cddf1a709ff671'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.617871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1aef935d95ac40ca82cddf1a709ff671'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.618564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1aef935d95ac40ca82cddf1a709ff671'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.619208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b567ef9a-4ad3-440b-866e-38cf73545e8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1aef935d95ac40ca82cddf1a709ff671'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:28.632134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1023/2025] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:28 2022] DELETE /identity/v3/users/1aef935d95ac40ca82cddf1a709ff671 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:28.927368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:28.929689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:28.958460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192cba3f-6832-49d9-ae17-81ca731c72c2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.053709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.054511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.054869 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.055284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] PATH_INFO: `/v3/projects/7b7b3e86685347cbb73d9e80036b45fa` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.056562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b7b3e86685347cbb73d9e80036b45fa)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.058623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.106121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b7b3e86685347cbb73d9e80036b45fa', 'request_id': 'req-a8019907-7179-4db8-9517-0e3c32c6d454'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:29.110499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:29.115516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b7b3e86685347cbb73d9e80036b45fa', 'request_id': 'req-a8019907-7179-4db8-9517-0e3c32c6d454'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:29.123570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b7b3e86685347cbb73d9e80036b45fa', 'request_id': 'req-a8019907-7179-4db8-9517-0e3c32c6d454'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:29.134183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] The token cache is being invalidate because project 7b7b3e86685347cbb73d9e80036b45fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:29.134582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:29.135292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a8019907-7179-4db8-9517-0e3c32c6d454 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:29.136776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1003/2026] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:28 2022] DELETE /identity/v3/projects/7b7b3e86685347cbb73d9e80036b45fa => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:29.141609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.142065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.142167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.163949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.165351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.223541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1024/2027] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.231535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.231830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.232269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.256488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.257993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.269415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1004/2028] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.273366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.284149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a3671d-19d2-4ee5-9248-be03c7bf5ec1 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.293731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.294428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.294586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.294860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.295512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.295822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.297582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.299276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:29.313429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1025/2029] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:29.317913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.327169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e37a4-6aa4-4bd4-a774-8e38f858287a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.331830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.332431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.332718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.333035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.333665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.335554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.363974 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1005/2030] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.368089 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.378051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2363ac-5f3b-43a8-8a91-75b94013a1fd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.432387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.433003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.433279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.433549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.434170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.436094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.456624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1026/2031] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/users => generated 293 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.460886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.471674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b93762e-3701-4491-bb4a-a9a87ad6cb8e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.477396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.478071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.478378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.478659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.479305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.480921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.493201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1006/2032] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:29.497314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.507709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05f5b6-2104-4442-a3bc-751ee3e61a34 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.512265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821b02e9-11fe-4e16-b5c4-cb3512d91967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.512851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b02e9-11fe-4e16-b5c4-cb3512d91967 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.513169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b02e9-11fe-4e16-b5c4-cb3512d91967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.513439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b02e9-11fe-4e16-b5c4-cb3512d91967 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.514093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821b02e9-11fe-4e16-b5c4-cb3512d91967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.515671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821b02e9-11fe-4e16-b5c4-cb3512d91967 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.526518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1027/2033] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:29.530567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.540617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181853e-e8aa-4ceb-90cd-811066945b97 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.545217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.545860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.546142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.546416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] PATH_INFO: `/v3/projects/3745efb7a10e434490dfab465c13c734/users/e3425435d5954e18b5ec99b0caf4b517/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.561298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3745efb7a10e434490dfab465c13c734, user_id=e3425435d5954e18b5ec99b0caf4b517, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.563963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.578321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1007/2034] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:29 2022] PUT /identity/v3/projects/3745efb7a10e434490dfab465c13c734/users/e3425435d5954e18b5ec99b0caf4b517/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:29.589846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.590136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.590416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.617895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] MFA Rules not processed for user `e3425435d5954e18b5ec99b0caf4b517`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.619125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.686759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1028/2035] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.692133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.751894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3af11b6-95c0-4526-bcc2-e135953e7822 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.757082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.757682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.757965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.758240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.758866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.760774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.786523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1008/2036] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.790727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.801741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a92481-fbd4-48c6-a21a-735214c947b1 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.867927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.868963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.869489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.869902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.870707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.872768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.886344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:29.911641 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1029/2037] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/users => generated 293 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:29.915669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.932466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafcb955-84ee-40eb-bbdb-a1dbd66c75cd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.937621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.938242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.938530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.938799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.939444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.941404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.955753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1009/2038] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:29.957000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.967324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e3307-0c1c-42d7-92a6-cf2c6634cde1 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.972569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b048ac-083f-47db-bde0-1b43031b0ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.973266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b048ac-083f-47db-bde0-1b43031b0ea3 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.973706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b048ac-083f-47db-bde0-1b43031b0ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.974038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b048ac-083f-47db-bde0-1b43031b0ea3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.974703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b048ac-083f-47db-bde0-1b43031b0ea3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.976460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b048ac-083f-47db-bde0-1b43031b0ea3 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.986351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1030/2039] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:29.991045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.002038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abdbcaf-7deb-4d12-a3ad-ffc7d83aa4b1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.006699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.007337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.007612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.007881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] PATH_INFO: `/v3/projects/aa8a836e1e9d4ff0962eeae107fdddb6/users/6ee20b6d021a4b2e8ac6366f82b9c991/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.023050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa8a836e1e9d4ff0962eeae107fdddb6, user_id=6ee20b6d021a4b2e8ac6366f82b9c991, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.025795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.038616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1010/2040] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:29 2022] PUT /identity/v3/projects/aa8a836e1e9d4ff0962eeae107fdddb6/users/6ee20b6d021a4b2e8ac6366f82b9c991/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:30.048962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.049258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.050120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.082408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] MFA Rules not processed for user `6ee20b6d021a4b2e8ac6366f82b9c991`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:30.084084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:30.156823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1031/2041] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:30.162306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.196684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:30.230960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b679cc-3140-4402-bce7-b918daeb6152 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.244104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.244896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.245189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.245463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.246161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.248057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.281151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1011/2042] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/projects => generated 385 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.282125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.297706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ce24c6-4332-4ac3-9e02-2c42ef6cc92b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.395196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.395936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.396319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.396797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.397825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.400016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.433398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1032/2043] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 299 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:30.437406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.451566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689b6825-914f-4397-83a7-7ff06217649c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.457803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.458479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.458765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.459606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.460350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.462103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.487646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.487754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1012/2044] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.502868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03e5753-f265-486b-bb21-25491ebcbee3 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.515821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.517131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.517556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.518458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] PATH_INFO: `/v3/projects/dc9dd2437f644851be8843ffb0914bb4/users/c7a97dc7510647d5969cff0a2bc729f9/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.554356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc9dd2437f644851be8843ffb0914bb4, user_id=c7a97dc7510647d5969cff0a2bc729f9, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.578735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.605277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1033/2045] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:30 2022] PUT /identity/v3/projects/dc9dd2437f644851be8843ffb0914bb4/users/c7a97dc7510647d5969cff0a2bc729f9/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:30.609317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.620588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83db89-b7ad-4631-b587-e2b6ac176ec9 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.683511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e3659e-8bd1-4ad6-bd39-d8b94bc70d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.684122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e3659e-8bd1-4ad6-bd39-d8b94bc70d11 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.684493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e3659e-8bd1-4ad6-bd39-d8b94bc70d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.684770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e3659e-8bd1-4ad6-bd39-d8b94bc70d11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.685479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e3659e-8bd1-4ad6-bd39-d8b94bc70d11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.687046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e3659e-8bd1-4ad6-bd39-d8b94bc70d11 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.697866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1013/2046] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.702132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.715584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a2abb5-6ac1-4a27-a70f-adf163f62eb2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.720563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.721860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.722204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.722537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] PATH_INFO: `/v3/projects/dc9dd2437f644851be8843ffb0914bb4/users/c7a97dc7510647d5969cff0a2bc729f9/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.724521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc9dd2437f644851be8843ffb0914bb4, user_id=c7a97dc7510647d5969cff0a2bc729f9, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.727202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.738578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1034/2047] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:30 2022] PUT /identity/v3/projects/dc9dd2437f644851be8843ffb0914bb4/users/c7a97dc7510647d5969cff0a2bc729f9/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:30.748565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.749053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.749334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.779252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] MFA Rules not processed for user `c7a97dc7510647d5969cff0a2bc729f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:30.780412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:30.854224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1014/2048] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:30.866319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.868220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:30.898786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:30.936567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecd64ec-5309-45f1-808f-6dc98084982f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.941473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.942186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.942454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.942721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.952978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.954481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.977775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1035/2049] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:31.735549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.746402 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f9a46c-7e7f-4b74-bb21-c411729f190a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.750033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0629c7f-5312-4e31-8357-a6d7c825333d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.750754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0629c7f-5312-4e31-8357-a6d7c825333d service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.751522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0629c7f-5312-4e31-8357-a6d7c825333d service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.751522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0629c7f-5312-4e31-8357-a6d7c825333d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.782854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:31.824908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0629c7f-5312-4e31-8357-a6d7c825333d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.826147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0629c7f-5312-4e31-8357-a6d7c825333d service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.837473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1015/2050] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:31.979540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.981276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:32.053929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef2ed82-656c-4ea6-aebd-cc6540da5d8a service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.058637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.059221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.059513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.059776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] PATH_INFO: `/v3/users/a800b175c9a9462089d86da9e226338c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.060997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a800b175c9a9462089d86da9e226338c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.062935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.121026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a800b175c9a9462089d86da9e226338c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.136486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a800b175c9a9462089d86da9e226338c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.145467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a800b175c9a9462089d86da9e226338c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.146535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a800b175c9a9462089d86da9e226338c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.147434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a800b175c9a9462089d86da9e226338c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.165491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1036/2051] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:31 2022] DELETE /identity/v3/users/a800b175c9a9462089d86da9e226338c => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:32.170092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.170430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.170708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.191077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:32.192507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:32.255468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1016/2052] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:32.266172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.267755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:32.313921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:32.326234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1892d4-4740-4e12-bfb4-12915e6679d7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.330244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.330888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.331165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.331433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.340227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.341613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.351335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1037/2053] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:32.639852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.642343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:32.671060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ea7523-c636-4c38-8ea0-587d86f58ad8 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.681222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.681970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.682329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.682676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] PATH_INFO: `/v3/projects/5fbf9c9a34c54fcfac32c229d3660449` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.684807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fbf9c9a34c54fcfac32c229d3660449)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.686947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.745773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fbf9c9a34c54fcfac32c229d3660449', 'request_id': 'req-5c2370a7-c728-47cc-b311-5b36dc036598'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.753205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fbf9c9a34c54fcfac32c229d3660449', 'request_id': 'req-5c2370a7-c728-47cc-b311-5b36dc036598'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.759730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fbf9c9a34c54fcfac32c229d3660449', 'request_id': 'req-5c2370a7-c728-47cc-b311-5b36dc036598'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.771322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] The token cache is being invalidate because project 5fbf9c9a34c54fcfac32c229d3660449 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:32.771671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.772331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.774196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1017/2054] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:32 2022] DELETE /identity/v3/projects/5fbf9c9a34c54fcfac32c229d3660449 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:32.779906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.790712 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81381406-ccb4-4bd2-ac70-6f214e5d8bac service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.858433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.859241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.859603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.860045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] PATH_INFO: `/v3/users/46623dbd007b4f82998d3919b7e9119e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.861490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=46623dbd007b4f82998d3919b7e9119e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.863554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.918505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46623dbd007b4f82998d3919b7e9119e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.932715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46623dbd007b4f82998d3919b7e9119e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.941520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46623dbd007b4f82998d3919b7e9119e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.942703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46623dbd007b4f82998d3919b7e9119e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.943761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46623dbd007b4f82998d3919b7e9119e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:32.960833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1038/2055] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:32 2022] DELETE /identity/v3/users/46623dbd007b4f82998d3919b7e9119e => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.795732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.797328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:33.811081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2370a7-c728-47cc-b311-5b36dc036598 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.862326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:33.880976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.880976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.880976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.881363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] PATH_INFO: `/v3/projects/64209f68dacc47139f3eed0ab0d00205` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.883564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64209f68dacc47139f3eed0ab0d00205)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.885758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.934933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64209f68dacc47139f3eed0ab0d00205', 'request_id': 'req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:33.943966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64209f68dacc47139f3eed0ab0d00205', 'request_id': 'req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:33.950915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64209f68dacc47139f3eed0ab0d00205', 'request_id': 'req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:33.962490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] The token cache is being invalidate because project 64209f68dacc47139f3eed0ab0d00205 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:33.962802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:33.963444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:33.964947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1018/2056] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:33 2022] DELETE /identity/v3/projects/64209f68dacc47139f3eed0ab0d00205 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.969063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.971552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:33.985034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8548705-8b53-41b2-826e-bc17c6e5d2bc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.048906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.050953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.053887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.054531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.054829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.055552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] PATH_INFO: `/v3/users/c73fe6208b544b8cbc7a2b2f8c9b0f5d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.056775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c73fe6208b544b8cbc7a2b2f8c9b0f5d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.059296 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.064369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd2cd9a-3a8b-47fa-b9a3-7d1ff2af13a9 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.067136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:34.071053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.071695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.072126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.072399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] PATH_INFO: `/v3/users/dea35bcb0d124a2ea034db0d5dd02393` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.073895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] RBAC: Authorizing `identity:delete_user(user_id=dea35bcb0d124a2ea034db0d5dd02393)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.075984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.129091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c73fe6208b544b8cbc7a2b2f8c9b0f5d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.145093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c73fe6208b544b8cbc7a2b2f8c9b0f5d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.149681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dea35bcb0d124a2ea034db0d5dd02393'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.153819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c73fe6208b544b8cbc7a2b2f8c9b0f5d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.154555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c73fe6208b544b8cbc7a2b2f8c9b0f5d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.155414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-52ffc961-29e8-4566-a862-7874654af5d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c73fe6208b544b8cbc7a2b2f8c9b0f5d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.158911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dea35bcb0d124a2ea034db0d5dd02393'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.160070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dea35bcb0d124a2ea034db0d5dd02393'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.161157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dea35bcb0d124a2ea034db0d5dd02393'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.170099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dea35bcb0d124a2ea034db0d5dd02393'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.175910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1039/2057] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:33 2022] DELETE /identity/v3/users/c73fe6208b544b8cbc7a2b2f8c9b0f5d => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:34.190663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1019/2058] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:34 2022] DELETE /identity/v3/users/dea35bcb0d124a2ea034db0d5dd02393 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:34.194835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.195150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.195423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.218515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:34.220160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.313343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1040/2059] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:34.323926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.325933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.409548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:34.418071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cfd0c8-1688-4ef3-b69d-2210786b171b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.423266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.425551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.425551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.425551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.436109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.437677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.449058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1020/2060] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:34.560350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.564408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.583660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81af215-2d07-41bd-b5a2-4f1a2660342c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.590082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.590852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.591204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.592479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] PATH_INFO: `/v3/users/f70dbcf39cdd49178867705333c094f7` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.593896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f70dbcf39cdd49178867705333c094f7)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.596774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.653060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.655303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.660918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f70dbcf39cdd49178867705333c094f7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.676237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f70dbcf39cdd49178867705333c094f7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.683593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f70dbcf39cdd49178867705333c094f7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.684616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f70dbcf39cdd49178867705333c094f7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.685444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c340bc5f-d975-4e07-8f26-0c7d33a71813 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f70dbcf39cdd49178867705333c094f7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.687789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a764859b-0111-4614-8e32-54d56ee8fd9a service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.700383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:34.701681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.702477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.702794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.703032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] PATH_INFO: `/v3/projects/754d0742bf9648e39239b72251c60fda` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.708018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1041/2061] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:34 2022] DELETE /identity/v3/users/f70dbcf39cdd49178867705333c094f7 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:34.709690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] RBAC: Authorizing `identity:delete_project(project_id=754d0742bf9648e39239b72251c60fda)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.713398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.713786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.714125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.723049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.763607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:34.765043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.836299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '754d0742bf9648e39239b72251c60fda', 'request_id': 'req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.847981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '754d0742bf9648e39239b72251c60fda', 'request_id': 'req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.857008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '754d0742bf9648e39239b72251c60fda', 'request_id': 'req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.871031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1042/2062] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:34.880099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] The token cache is being invalidate because project 754d0742bf9648e39239b72251c60fda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:34.881267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.883042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.885303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.889071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a471a511-5f2f-41ab-ba7b-9be5a9e9531f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.891666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1021/2063] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:34 2022] DELETE /identity/v3/projects/754d0742bf9648e39239b72251c60fda => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:34.902513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970356c5-8a79-433e-993b-888f1de6bdd5 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.902954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970356c5-8a79-433e-993b-888f1de6bdd5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.903381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970356c5-8a79-433e-993b-888f1de6bdd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.948129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-970356c5-8a79-433e-993b-888f1de6bdd5 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:34.951768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-970356c5-8a79-433e-993b-888f1de6bdd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.001520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6097c1d4-aa0a-4d5d-8489-2f4eaea4e3be None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.045068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1022/2064] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:35.058608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.058991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.059421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.088427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.091283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.091686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.092243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.097279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.099365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:35.100568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.114579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1023/2065] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:35.115635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.118594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.126315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.132098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.134355 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb91b01d-329f-4153-9950-06eab1be6f95 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.141259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.141943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.142294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.142672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] PATH_INFO: `/v3/projects/968d4393b70f4260aea95f8860ed24cc` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.143911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=968d4393b70f4260aea95f8860ed24cc)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.146225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.155054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1043/2066] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:35.157672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.207631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8e634d-a765-454e-bf7e-15b9175067dc service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.235335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.236206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.236709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.237176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.237987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:35.238433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.243555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.245273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '968d4393b70f4260aea95f8860ed24cc', 'request_id': 'req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.255682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '968d4393b70f4260aea95f8860ed24cc', 'request_id': 'req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.268093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '968d4393b70f4260aea95f8860ed24cc', 'request_id': 'req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.276093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:35.285317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1044/2067] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:35.286367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.314164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e423e6-e89e-4675-a8a0-eb6811d6b82b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.319541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] The token cache is being invalidate because project 968d4393b70f4260aea95f8860ed24cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:35.320020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.321229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.323718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1024/2068] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:35 2022] DELETE /identity/v3/projects/968d4393b70f4260aea95f8860ed24cc => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:35.327794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.329436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.380043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:35.398355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.398617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.398952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.399455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.400071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.402405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.443952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1045/2069] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:35 2022] POST /identity/v3/projects => generated 373 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:35.444272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137e4a96-7e7e-4ba6-9e50-aaf4bcf267a1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.445888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.448347 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.559435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6716de63-245c-4d65-846c-f9c17a14a56a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.590064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.590330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.590722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.591002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.591277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] PATH_INFO: `/v3/users/54da39f14bd34e6996e6ae4067c39300` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.591651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.592529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] RBAC: Authorizing `identity:delete_user(user_id=54da39f14bd34e6996e6ae4067c39300)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.593004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.593425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.595234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.596657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.601749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.638062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1046/2070] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:35 2022] POST /identity/v3/users => generated 287 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:35.642205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.664567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c5e7e-d243-4ea1-abc2-53cbf51b6168 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.673440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.674946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.675398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.675819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.681284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54da39f14bd34e6996e6ae4067c39300'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.683398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.687495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.707007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1047/2071] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:35.711621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.715930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54da39f14bd34e6996e6ae4067c39300'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.718209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54da39f14bd34e6996e6ae4067c39300'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.720685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54da39f14bd34e6996e6ae4067c39300'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.748088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9548f162-074c-421f-9c30-816b383af006 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.754475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.755386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.755962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.756391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.757432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.758010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54da39f14bd34e6996e6ae4067c39300'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:35.760533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.782507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1048/2072] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:35.784600 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.787003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.796857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1025/2073] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:35 2022] DELETE /identity/v3/users/54da39f14bd34e6996e6ae4067c39300 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:35.798166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:35.799697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:35.822788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:35.823739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bea4b8e-8508-438d-be65-65b2c8de4631 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.931921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.932657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8929c7f-3551-4e41-9856-11ecf8489aec admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:35.933359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.934410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.934837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] PATH_INFO: `/v3/projects/8690cde79c8844b1a57f318293d0f5f5` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.936229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8690cde79c8844b1a57f318293d0f5f5)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.938872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:35.945498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c58c1a-461e-41ee-a408-0e9d62dd0461 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:35.947238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c58c1a-461e-41ee-a408-0e9d62dd0461 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:35.947772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c58c1a-461e-41ee-a408-0e9d62dd0461 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:35.949202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c58c1a-461e-41ee-a408-0e9d62dd0461 admin admin] PATH_INFO: `/v3/projects/ce850c7c7ff34635b95a5caea9b83baa/users/a5a6fefa82e64e64a7e93d8295ceddc8/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:35.977285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c58c1a-461e-41ee-a408-0e9d62dd0461 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce850c7c7ff34635b95a5caea9b83baa, user_id=a5a6fefa82e64e64a7e93d8295ceddc8, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:35.980557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c58c1a-461e-41ee-a408-0e9d62dd0461 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.016165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1026/2074] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:35 2022] PUT /identity/v3/projects/ce850c7c7ff34635b95a5caea9b83baa/users/a5a6fefa82e64e64a7e93d8295ceddc8/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:36.018266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.018555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.018900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.046369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:36.057225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8690cde79c8844b1a57f318293d0f5f5', 'request_id': 'req-e5e8ef58-ab9b-462f-b927-ac4095c595c1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.078761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] MFA Rules not processed for user `a5a6fefa82e64e64a7e93d8295ceddc8`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:36.081280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8690cde79c8844b1a57f318293d0f5f5', 'request_id': 'req-e5e8ef58-ab9b-462f-b927-ac4095c595c1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.085388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.091588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8690cde79c8844b1a57f318293d0f5f5', 'request_id': 'req-e5e8ef58-ab9b-462f-b927-ac4095c595c1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.131351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] The token cache is being invalidate because project 8690cde79c8844b1a57f318293d0f5f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:36.131775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.134110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.137150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1049/2075] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:35 2022] DELETE /identity/v3/projects/8690cde79c8844b1a57f318293d0f5f5 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:36.141866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.144781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.268745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1027/2076] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:36.279975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.282553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.324261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e8ef58-ab9b-462f-b927-ac4095c595c1 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:36.342378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:36.344368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.344743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.347161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] PATH_INFO: `/v3/users/d7d10dcf130643b780bac8d257d39fe1` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.350581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7d10dcf130643b780bac8d257d39fe1)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:36.355735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.443770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ceed9-d7f1-4419-adfb-d46299ef1c6b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:36.450520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:36.451541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.451935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.452309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.464333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:36.466809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.479089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1028/2077] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:36.480305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.482069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.499538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df0252e-924a-414a-8ff6-095fc4d96cef service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:36.505308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:36.505445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7d10dcf130643b780bac8d257d39fe1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.506019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.506227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.506540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.508979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.530019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7d10dcf130643b780bac8d257d39fe1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.551324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7d10dcf130643b780bac8d257d39fe1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.552178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7d10dcf130643b780bac8d257d39fe1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.552898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7d10dcf130643b780bac8d257d39fe1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.570845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:36.591768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1050/2078] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:36 2022] DELETE /identity/v3/users/d7d10dcf130643b780bac8d257d39fe1 => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:36.601344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.604340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.640117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:36.641653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.666944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1029/2079] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:36.697511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:36.724739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ad72a1-73f9-444a-8076-0ade8c7befba admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:36.739395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:36.740782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.741105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.741778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] PATH_INFO: `/v3/projects/729aec6772cb4365aa3c801ccf482a5c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.743586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] RBAC: Authorizing `identity:delete_project(project_id=729aec6772cb4365aa3c801ccf482a5c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:36.746942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.844687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '729aec6772cb4365aa3c801ccf482a5c', 'request_id': 'req-fed7e6f3-6cb0-407c-93e9-6b649054029d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.857926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '729aec6772cb4365aa3c801ccf482a5c', 'request_id': 'req-fed7e6f3-6cb0-407c-93e9-6b649054029d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.870592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '729aec6772cb4365aa3c801ccf482a5c', 'request_id': 'req-fed7e6f3-6cb0-407c-93e9-6b649054029d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.908523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] The token cache is being invalidate because project 729aec6772cb4365aa3c801ccf482a5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:36.908939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.909850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.911548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1051/2080] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:36 2022] DELETE /identity/v3/projects/729aec6772cb4365aa3c801ccf482a5c => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:36.915536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.917086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.933193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9982e-a49d-4375-b614-40d2587d4c6c service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.003193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.004161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.004525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.004859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] PATH_INFO: `/v3/users/5dcc9ed5050144bd81a6841728f8b13c` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.006173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dcc9ed5050144bd81a6841728f8b13c)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.008377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.092704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dcc9ed5050144bd81a6841728f8b13c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.105923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dcc9ed5050144bd81a6841728f8b13c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.106938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dcc9ed5050144bd81a6841728f8b13c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.108801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dcc9ed5050144bd81a6841728f8b13c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.135555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dcc9ed5050144bd81a6841728f8b13c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.157406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1030/2081] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:36 2022] DELETE /identity/v3/users/5dcc9ed5050144bd81a6841728f8b13c => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:37.306163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.307691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:37.322504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed7e6f3-6cb0-407c-93e9-6b649054029d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.385998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.386636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.386911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.387181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] PATH_INFO: `/v3/projects/4482eb58d6fe46eb970a64dc2f834e3a` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.388424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4482eb58d6fe46eb970a64dc2f834e3a)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.390357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.437947 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4482eb58d6fe46eb970a64dc2f834e3a', 'request_id': 'req-8059382f-f7c2-4e5d-8cc6-28c96718d24a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.447231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4482eb58d6fe46eb970a64dc2f834e3a', 'request_id': 'req-8059382f-f7c2-4e5d-8cc6-28c96718d24a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.453416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4482eb58d6fe46eb970a64dc2f834e3a', 'request_id': 'req-8059382f-f7c2-4e5d-8cc6-28c96718d24a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.464560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.465170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] The token cache is being invalidate because project 4482eb58d6fe46eb970a64dc2f834e3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:37.465460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.466104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:37.466217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8059382f-f7c2-4e5d-8cc6-28c96718d24a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.467584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1052/2082] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:37 2022] DELETE /identity/v3/projects/4482eb58d6fe46eb970a64dc2f834e3a => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:37.473482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec47e721-4893-42a8-a28b-07db34e16efd None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.473766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec47e721-4893-42a8-a28b-07db34e16efd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.474099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec47e721-4893-42a8-a28b-07db34e16efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.480252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44094e2-3e23-4a71-b743-ff16e5eda1b8 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.492954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:37.499202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ec47e721-4893-42a8-a28b-07db34e16efd None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:37.500626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ec47e721-4893-42a8-a28b-07db34e16efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:37.533162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:37.555468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.556107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.556484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.556800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] PATH_INFO: `/v3/projects/9694c99c58ee4d60b40edb6e9fc93102` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.558036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9694c99c58ee4d60b40edb6e9fc93102)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.560060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.576855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1053/2083] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:37.584935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.585224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.585498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.607067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:37.608321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:37.615088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9694c99c58ee4d60b40edb6e9fc93102', 'request_id': 'req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.622757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9694c99c58ee4d60b40edb6e9fc93102', 'request_id': 'req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.629835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9694c99c58ee4d60b40edb6e9fc93102', 'request_id': 'req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.641684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] The token cache is being invalidate because project 9694c99c58ee4d60b40edb6e9fc93102 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:37.642255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.643126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5ecd8342-a083-4374-89dc-7cc5cb22c4e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:37.644654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1031/2084] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:37 2022] DELETE /identity/v3/projects/9694c99c58ee4d60b40edb6e9fc93102 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:37.649632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eadd52-9bd7-4958-9eb4-cf9c4ad216f8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.649964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eadd52-9bd7-4958-9eb4-cf9c4ad216f8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.650240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eadd52-9bd7-4958-9eb4-cf9c4ad216f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.670739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c6eadd52-9bd7-4958-9eb4-cf9c4ad216f8 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:37.670937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1054/2085] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:37.672033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c6eadd52-9bd7-4958-9eb4-cf9c4ad216f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:37.675253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.680273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1032/2086] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:37.684391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e765f7-be86-47f6-9500-039b30c09c8e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.688626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.688626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.689014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.694378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.695106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.695453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.695794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.696513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:37.696899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.698678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.710745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1055/2087] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:12:37.711352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:37.712648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:37.714826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.721477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1033/2088] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:37.725223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd24710-c861-425e-acfa-926e9f6e0114 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.725614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.729875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.730460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.730740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.731012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.731633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.733561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.735155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489ada33-0e04-4c67-a650-04ff2237f368 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.739669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.740450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.740619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.740777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.741448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:37.741747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.743305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.759835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1034/2089] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:12:37.760986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.764990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1056/2090] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:37.766005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.771357 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e4a8cd-e56d-4f6a-b7a4-e1ba7cb81261 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.836153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.836744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.837074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.837352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.838257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.838692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e3bc0c-58d8-4175-9581-fc9977bd1f3c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.842681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.843713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.844829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.844972 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.845276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.846516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.848546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.880584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1035/2091] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/projects => generated 377 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:37.880788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1057/2092] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/users => generated 283 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:37.880907 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.882682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:37.889857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:37.897552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0075a71-a183-4c0f-8e04-318456d0c7b7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.945617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:37.986742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9ccc5-bad2-4923-9f2d-af71652df140 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:37.992156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.992614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:37.992802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.993251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.993387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:37.993533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.993669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:37.993943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:37.994183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.994797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:37.995859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:37.996714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.014633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1058/2093] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:37 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:38.019663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.023408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1036/2094] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:37 2022] POST /identity/v3/users => generated 289 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:38.029751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.032612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8089d4a-d7a0-4aee-be3a-b702893c633a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.039993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.040586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.040894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.041174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.041818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.042015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb5f236-809e-4ec1-866d-b3f4284e6d26 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.043619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.047276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.047882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.048161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.048437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.050069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.051661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.055798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1059/2095] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:38.060603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.062751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1037/2096] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:38.067065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.071575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a212da-b53a-4047-8ce6-6d7b64c55ebc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.077002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a09ca7-8348-41ee-bfc4-dd3c8b31f05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.077480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33116851-9d9d-4e72-8b86-01346538f46b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.077653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a09ca7-8348-41ee-bfc4-dd3c8b31f05b admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.077935 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a09ca7-8348-41ee-bfc4-dd3c8b31f05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.078210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a09ca7-8348-41ee-bfc4-dd3c8b31f05b admin admin] PATH_INFO: `/v3/projects/a615871ac78643768889b080951a8dd5/users/463301a85335492e98c99c40bf45cd21/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.082917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.083467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.083744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.084017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.084678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.086346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.095159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a09ca7-8348-41ee-bfc4-dd3c8b31f05b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a615871ac78643768889b080951a8dd5, user_id=463301a85335492e98c99c40bf45cd21, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.098001 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a09ca7-8348-41ee-bfc4-dd3c8b31f05b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.101173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1038/2097] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:38.106182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.113953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1060/2098] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:38 2022] PUT /identity/v3/projects/a615871ac78643768889b080951a8dd5/users/463301a85335492e98c99c40bf45cd21/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:38.118400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08189b2-1e97-4a52-8167-fd1c310dc8e1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.125265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.125556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.125830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.159778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] MFA Rules not processed for user `463301a85335492e98c99c40bf45cd21`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:38.161143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:38.192657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151280a8-a55a-4fb7-bca4-361cac288f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.193320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151280a8-a55a-4fb7-bca4-361cac288f38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.193603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151280a8-a55a-4fb7-bca4-361cac288f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.193887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151280a8-a55a-4fb7-bca4-361cac288f38 admin admin] PATH_INFO: `/v3/projects/165a7d71a21240a79e77abbe80717389/users/b38384d843d04503b69e5cb61af6c6ec/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.224768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151280a8-a55a-4fb7-bca4-361cac288f38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=165a7d71a21240a79e77abbe80717389, user_id=b38384d843d04503b69e5cb61af6c6ec, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.232585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151280a8-a55a-4fb7-bca4-361cac288f38 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.258300 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1039/2099] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:38 2022] PUT /identity/v3/projects/165a7d71a21240a79e77abbe80717389/users/b38384d843d04503b69e5cb61af6c6ec/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:38.265512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.265818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.268687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.271853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1061/2100] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:38 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:38.277193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.294526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a93651a-321b-4a6e-b3c6-13a8fe5b3d6c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.309684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:38.312216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:38.409476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:38.411620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1040/2101] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:38.417121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.418289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.418574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.418848 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.420857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.422416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.425994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.426126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:38.460898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1062/2102] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:38 2022] POST /identity/v3/projects => generated 365 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:38.465492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.475961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:38.515870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24fb142-19cb-4a41-b723-1929fb5212ef None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.560275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eb1542-4de9-4659-be80-0718fd0c9414 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.567432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.568367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.568651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.569005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.569643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.580966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.597347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.597963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.598244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.598708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.604933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1063/2103] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:38 2022] POST /identity/v3/users => generated 289 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:38.609208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.610299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.611984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.623194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a104375c-5196-49a3-993d-d028a8bcbc7a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.625240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1041/2104] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:38.633681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.634399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.634931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.635290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.636026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.637946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.673209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1064/2105] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:38.673499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.685100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29709a97-029a-44f8-bf1b-1904e50fa14d service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.694315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.695895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.696422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.696841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] PATH_INFO: `/v3/projects/c9a8f110186b4b7e9137fa82794786f6/users/2ffc604863374ec0a5c7e9226ba4cc33/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.720704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9a8f110186b4b7e9137fa82794786f6, user_id=2ffc604863374ec0a5c7e9226ba4cc33, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.727098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.764324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1042/2106] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:38 2022] PUT /identity/v3/projects/c9a8f110186b4b7e9137fa82794786f6/users/2ffc604863374ec0a5c7e9226ba4cc33/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:38.769632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.784027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b293091-4f7a-435c-8092-238e45ec964c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.845470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336b11c9-6951-415f-9a89-7d3c35048fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.846523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336b11c9-6951-415f-9a89-7d3c35048fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.846727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336b11c9-6951-415f-9a89-7d3c35048fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.847422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336b11c9-6951-415f-9a89-7d3c35048fa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.848735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336b11c9-6951-415f-9a89-7d3c35048fa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.850350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336b11c9-6951-415f-9a89-7d3c35048fa7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.866662 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1065/2107] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:38 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:38.867739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.879240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc0b532-a7c2-41ff-844f-7b61a956f11e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.884251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.884933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.885219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.885494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] PATH_INFO: `/v3/projects/c9a8f110186b4b7e9137fa82794786f6/users/2ffc604863374ec0a5c7e9226ba4cc33/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.887539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9a8f110186b4b7e9137fa82794786f6, user_id=2ffc604863374ec0a5c7e9226ba4cc33, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:38.890455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:38.901571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1043/2108] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:38 2022] PUT /identity/v3/projects/c9a8f110186b4b7e9137fa82794786f6/users/2ffc604863374ec0a5c7e9226ba4cc33/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:38.912616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.912929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.913206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.942615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] MFA Rules not processed for user `2ffc604863374ec0a5c7e9226ba4cc33`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:38.944501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:39.004252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:39.051605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1066/2109] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:38 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:39.065839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:39.067877 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:39.155935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520e6b6-64a4-47af-a51d-886589074216 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:39.161306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:39.161621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.161905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.162184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.173865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.175594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.193714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1044/2110] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:39 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:39.276819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:39.286798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c714770c-c006-466f-b600-8729b9e0cc35 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:39.290278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c613bc-f51b-410c-a4cb-e49ef4bd650e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:39.290871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c613bc-f51b-410c-a4cb-e49ef4bd650e service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.291146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c613bc-f51b-410c-a4cb-e49ef4bd650e service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.291421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c613bc-f51b-410c-a4cb-e49ef4bd650e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.348251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c613bc-f51b-410c-a4cb-e49ef4bd650e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.349626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c613bc-f51b-410c-a4cb-e49ef4bd650e service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.358870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1067/2111] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:39 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:40.692160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.693717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:40.800285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:40.807808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60bc47c-3513-4a66-b695-807adea500b0 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.816587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.817399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.817758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.818414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] PATH_INFO: `/v3/users/e3425435d5954e18b5ec99b0caf4b517` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.819929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3425435d5954e18b5ec99b0caf4b517)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:40.829063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:40.997908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3425435d5954e18b5ec99b0caf4b517'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.009497 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3425435d5954e18b5ec99b0caf4b517'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.010744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3425435d5954e18b5ec99b0caf4b517'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.042036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3425435d5954e18b5ec99b0caf4b517'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.061630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3425435d5954e18b5ec99b0caf4b517'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.108689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1045/2112] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:40 2022] DELETE /identity/v3/users/e3425435d5954e18b5ec99b0caf4b517 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:41.109463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.109918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.110008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.131523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:41.132757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:41.211518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1068/2113] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:41.222143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.224238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:41.333308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe67f5a-a33d-4e56-a88f-24a9fa30f334 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:41.337515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:41.338132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.338410 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.338678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.348428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:41.350099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:41.371712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1046/2114] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:41.765680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.767276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:41.802675 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38698f01-d01d-42e4-9283-49d7285a2882 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:41.808905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:41.809780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.810130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.810476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] PATH_INFO: `/v3/projects/3745efb7a10e434490dfab465c13c734` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.812041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3745efb7a10e434490dfab465c13c734)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:41.814201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:41.904245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3745efb7a10e434490dfab465c13c734', 'request_id': 'req-7258c996-5ae5-4079-98e6-09ee8619991a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.916455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3745efb7a10e434490dfab465c13c734', 'request_id': 'req-7258c996-5ae5-4079-98e6-09ee8619991a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.923020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3745efb7a10e434490dfab465c13c734', 'request_id': 'req-7258c996-5ae5-4079-98e6-09ee8619991a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.933956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:41.940959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] The token cache is being invalidate because project 3745efb7a10e434490dfab465c13c734 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:41.941586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.953398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:41.953398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1069/2115] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:41 2022] DELETE /identity/v3/projects/3745efb7a10e434490dfab465c13c734 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:41.956177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.966904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933efdc-dd52-4db9-a267-e35711fc0e7d service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:42.068443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:42.069353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:42.069872 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:42.070373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] PATH_INFO: `/v3/users/6ee20b6d021a4b2e8ac6366f82b9c991` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:42.073167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ee20b6d021a4b2e8ac6366f82b9c991)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:42.075558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:42.125955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:42.179797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ee20b6d021a4b2e8ac6366f82b9c991'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.197580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ee20b6d021a4b2e8ac6366f82b9c991'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.198351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ee20b6d021a4b2e8ac6366f82b9c991'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.198973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ee20b6d021a4b2e8ac6366f82b9c991'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.213965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ee20b6d021a4b2e8ac6366f82b9c991'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.250346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1047/2116] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:41 2022] DELETE /identity/v3/users/6ee20b6d021a4b2e8ac6366f82b9c991 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:42.545108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:42.547182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:42.574728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7258c996-5ae5-4079-98e6-09ee8619991a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:42.754400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:42.755319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:42.755797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:42.756281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] PATH_INFO: `/v3/users/463301a85335492e98c99c40bf45cd21` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:42.757733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] RBAC: Authorizing `identity:delete_user(user_id=463301a85335492e98c99c40bf45cd21)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:42.759880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:42.836851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '463301a85335492e98c99c40bf45cd21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.885107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:42.885107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:42.898637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '463301a85335492e98c99c40bf45cd21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.908042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6d7f36-ed3f-4455-add5-6a9be5dca8f0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:42.920593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '463301a85335492e98c99c40bf45cd21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.920820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:42.921541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '463301a85335492e98c99c40bf45cd21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.922199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:42.922484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:42.922790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] PATH_INFO: `/v3/projects/aa8a836e1e9d4ff0962eeae107fdddb6` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:42.926521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-4594b759-ede5-4f72-b6b3-62df95d6f895 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '463301a85335492e98c99c40bf45cd21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:42.926743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa8a836e1e9d4ff0962eeae107fdddb6)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:42.932666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:42.967219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1070/2117] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:42 2022] DELETE /identity/v3/users/463301a85335492e98c99c40bf45cd21 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:42.968805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:42.969116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:42.969560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.014446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:43.017443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:43.109505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa8a836e1e9d4ff0962eeae107fdddb6', 'request_id': 'req-7c3808e8-d378-464c-a6f7-0f62e7052a7e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.148507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa8a836e1e9d4ff0962eeae107fdddb6', 'request_id': 'req-7c3808e8-d378-464c-a6f7-0f62e7052a7e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.150531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1071/2118] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:43.174791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:43.176611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:43.181374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa8a836e1e9d4ff0962eeae107fdddb6', 'request_id': 'req-7c3808e8-d378-464c-a6f7-0f62e7052a7e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.217008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:43.221958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] The token cache is being invalidate because project aa8a836e1e9d4ff0962eeae107fdddb6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:43.222256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.223894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.226419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1048/2119] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:42 2022] DELETE /identity/v3/projects/aa8a836e1e9d4ff0962eeae107fdddb6 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:43.238775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:43.245545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:43.366373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0884b6a8-bdb8-4667-b80e-24c80740ba8b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:43.464400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3808e8-d378-464c-a6f7-0f62e7052a7e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:43.475996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:43.476918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:43.477068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:43.477807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:43.478338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.481301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:43.484413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:43.484729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:43.485046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] PATH_INFO: `/v3/users/c7a97dc7510647d5969cff0a2bc729f9` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.489936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7a97dc7510647d5969cff0a2bc729f9)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:43.494677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:43.516794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:43.520246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:43.536441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1072/2120] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:43.574397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:43.634539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7a97dc7510647d5969cff0a2bc729f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.656475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7a97dc7510647d5969cff0a2bc729f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.657301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7a97dc7510647d5969cff0a2bc729f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.658057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7a97dc7510647d5969cff0a2bc729f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.668800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7a97dc7510647d5969cff0a2bc729f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:43.709392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1049/2121] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:43 2022] DELETE /identity/v3/users/c7a97dc7510647d5969cff0a2bc729f9 => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:44.445320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:44.467252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:44.504274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7e704-d5b8-4138-bd32-bf7469247e0c service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:44.514650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:44.517138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:44.665623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eb8e4d-9ea9-4558-a458-3486da3f5c47 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:44.670688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:44.672557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:44.672557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:44.672557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] PATH_INFO: `/v3/projects/dc9dd2437f644851be8843ffb0914bb4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:44.675386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc9dd2437f644851be8843ffb0914bb4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:44.675592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:44.741095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:44.742128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:44.742642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:44.743119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] PATH_INFO: `/v3/projects/a615871ac78643768889b080951a8dd5` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:44.746933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a615871ac78643768889b080951a8dd5)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:44.763286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:44.803353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc9dd2437f644851be8843ffb0914bb4', 'request_id': 'req-9e46f66d-bd97-4998-aa28-aa7164107372'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.822339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc9dd2437f644851be8843ffb0914bb4', 'request_id': 'req-9e46f66d-bd97-4998-aa28-aa7164107372'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.838335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc9dd2437f644851be8843ffb0914bb4', 'request_id': 'req-9e46f66d-bd97-4998-aa28-aa7164107372'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.854165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] The token cache is being invalidate because project dc9dd2437f644851be8843ffb0914bb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:44.854702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.855718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e46f66d-bd97-4998-aa28-aa7164107372 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.857549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1050/2122] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:44 2022] DELETE /identity/v3/projects/dc9dd2437f644851be8843ffb0914bb4 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:44.899161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45830eb0-cc01-4c80-89d5-5421946e79d6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:44.901105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45830eb0-cc01-4c80-89d5-5421946e79d6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:44.901874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45830eb0-cc01-4c80-89d5-5421946e79d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:44.957152 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a615871ac78643768889b080951a8dd5', 'request_id': 'req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.972433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-45830eb0-cc01-4c80-89d5-5421946e79d6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:44.974269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-45830eb0-cc01-4c80-89d5-5421946e79d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:44.981886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a615871ac78643768889b080951a8dd5', 'request_id': 'req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:44.988981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a615871ac78643768889b080951a8dd5', 'request_id': 'req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.006031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:45.017733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] The token cache is being invalidate because project a615871ac78643768889b080951a8dd5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:45.018109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.019577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.022684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1073/2123] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:44 2022] DELETE /identity/v3/projects/a615871ac78643768889b080951a8dd5 => generated 0 bytes in 578 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:45.026730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.032525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:45.062270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1051/2124] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:45.063908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.064266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.064512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.076156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8ce6e-d355-4155-a4a3-651a442a1f2f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.092540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:45.092742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.093263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.093598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.093882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] PATH_INFO: `/v3/users/2ffc604863374ec0a5c7e9226ba4cc33` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.095388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:45.102603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ffc604863374ec0a5c7e9226ba4cc33)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.104579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.113229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1052/2125] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:45.114374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.115027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:45.128374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d4f693-f806-4b6e-b277-643127a3f469 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.133542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.134554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.135086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.135524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.136319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:45.136766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.138533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.157846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1053/2126] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:45.160345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.186564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ea1f71-f042-4763-884f-10805d4b9c71 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.199979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.201421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.202071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.202173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.208585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.211888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.243663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ffc604863374ec0a5c7e9226ba4cc33'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.277421 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ffc604863374ec0a5c7e9226ba4cc33'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.289677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1054/2127] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/projects => generated 381 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:45.290695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.295262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ffc604863374ec0a5c7e9226ba4cc33'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.298547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ffc604863374ec0a5c7e9226ba4cc33'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.299867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0ce52f91-726f-4e83-9339-3a5f138aa262 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ffc604863374ec0a5c7e9226ba4cc33'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:45.327988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1074/2128] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:45 2022] DELETE /identity/v3/users/2ffc604863374ec0a5c7e9226ba4cc33 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:45.390942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c38cbd-eedb-4348-b3c8-0e2cedb9981a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.529102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.530122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.530483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.568257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] MFA Rules not processed for user `b38384d843d04503b69e5cb61af6c6ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:45.570183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:45.573845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.575409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.575920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.576373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.577308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.579720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.626502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1055/2129] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/users => generated 291 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:45.630892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.636494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:45.663331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bea1f-6cba-4c17-81c2-09dd11f382a4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.670036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.670677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.670934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.671197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.671997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.673444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.686149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1056/2130] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:45 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:45.691680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.734585 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bc24e3-a95c-4b65-bfa1-913f2637cb7f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.744916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.745543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.745823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.746101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.747131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.748452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.758829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1075/2131] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:45.774645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1057/2132] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:45 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:45.784980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.785141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.790501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:45.804784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:45.819060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544445eb-45d8-4579-b91f-d9d905e29600 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.829070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d085e7a-d990-4d4c-847d-0f72a8c91626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.829759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d085e7a-d990-4d4c-847d-0f72a8c91626 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.830104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d085e7a-d990-4d4c-847d-0f72a8c91626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.830385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d085e7a-d990-4d4c-847d-0f72a8c91626 admin admin] PATH_INFO: `/v3/projects/3eab3c9cf27349c5ab214851e53938c1/users/a2ce06d6850f48f581d61be4b7377c26/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.837548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faab44bc-25e0-4160-a2dc-720e8f6cde91 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:45.842456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:45.843243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.848444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.848817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.849527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.852671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.854414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d085e7a-d990-4d4c-847d-0f72a8c91626 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3eab3c9cf27349c5ab214851e53938c1, user_id=a2ce06d6850f48f581d61be4b7377c26, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:45.857094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d085e7a-d990-4d4c-847d-0f72a8c91626 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:45.881861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1058/2133] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:45 2022] PUT /identity/v3/projects/3eab3c9cf27349c5ab214851e53938c1/users/a2ce06d6850f48f581d61be4b7377c26/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:45.894060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.894469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.894847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.905418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1076/2134] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/projects => generated 379 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:45.906532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:45.943588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] MFA Rules not processed for user `a2ce06d6850f48f581d61be4b7377c26`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:45.949882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.007179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:46.047216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9acbff-1bb9-426a-b274-8c52f2c61e87 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.057680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.058379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.058715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.059046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.059685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.061769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.107663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1077/2135] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/users => generated 296 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:46.108375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.119957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1059/2136] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.123144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.124492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.142392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f3f60-6f9d-4a33-bc00-57f7488fb9aa None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.153246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d4b052-fab6-45b3-8af5-901a7156a94b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.154491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.155015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.155295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.155571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.156252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.157898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.159169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.160132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.160938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.161360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] PATH_INFO: `/v3/projects/c9a8f110186b4b7e9137fa82794786f6` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.162616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9a8f110186b4b7e9137fa82794786f6)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.165597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.176412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1060/2137] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:46.176412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.197806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b150f-6cca-42f3-b165-0fd43feb31f2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.210520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.211145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.211426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.213162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.213828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.218978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.246399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9a8f110186b4b7e9137fa82794786f6', 'request_id': 'req-a19a1a80-4cf8-45f9-952a-4634e4ce8985'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:46.261880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9a8f110186b4b7e9137fa82794786f6', 'request_id': 'req-a19a1a80-4cf8-45f9-952a-4634e4ce8985'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:46.274866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9a8f110186b4b7e9137fa82794786f6', 'request_id': 'req-a19a1a80-4cf8-45f9-952a-4634e4ce8985'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:46.284979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1061/2138] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/projects => generated 381 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:46.284979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.298384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] The token cache is being invalidate because project c9a8f110186b4b7e9137fa82794786f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:46.299487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:46.300237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:46.302378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1078/2139] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:46 2022] DELETE /identity/v3/projects/c9a8f110186b4b7e9137fa82794786f6 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:46.303053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.306509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.424272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74717ec-d572-4f95-be1b-d2350b64ea05 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.443667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb5817e-20e4-4968-ab43-ffa7dfc0972b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.444646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5817e-20e4-4968-ab43-ffa7dfc0972b admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.445140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5817e-20e4-4968-ab43-ffa7dfc0972b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.445567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5817e-20e4-4968-ab43-ffa7dfc0972b admin admin] PATH_INFO: `/v3/projects/47aa1e9c32034f62b9c720cc48ec7338/users/96c7129b47e94199bd8975bd3d3c98c7/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.454885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a1a80-4cf8-45f9-952a-4634e4ce8985 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.461796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.462875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.463470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.463851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.464661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.467409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.478020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb5817e-20e4-4968-ab43-ffa7dfc0972b admin admin] RBAC: Authorizing `identity:create_grant(project_id=47aa1e9c32034f62b9c720cc48ec7338, user_id=96c7129b47e94199bd8975bd3d3c98c7, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.481341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb5817e-20e4-4968-ab43-ffa7dfc0972b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.507749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1062/2140] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:46 2022] PUT /identity/v3/projects/47aa1e9c32034f62b9c720cc48ec7338/users/96c7129b47e94199bd8975bd3d3c98c7/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:46.509480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.510280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.510572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.515546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1079/2141] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/users => generated 297 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:46.516499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.521602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.538920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250266c-749f-45c0-992e-781ae9e39e5a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.547269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:46.549128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.572607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:46.659590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=86,keystone:INSERT=7,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:46.687038 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e71ce0-6835-4f4f-a3e6-f2f575917b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.687682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e71ce0-6835-4f4f-a3e6-f2f575917b9d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.688366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e71ce0-6835-4f4f-a3e6-f2f575917b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.688710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e71ce0-6835-4f4f-a3e6-f2f575917b9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.690158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e71ce0-6835-4f4f-a3e6-f2f575917b9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.701223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e71ce0-6835-4f4f-a3e6-f2f575917b9d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.702036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1063/2142] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.703503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.713638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846ed16-bc47-4e8a-b8ab-3db16c03ca28 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.724113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1080/2143] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:46.725476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.725763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.727143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.727439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.728068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.729748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.730255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.731474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.733602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.752776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1064/2144] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:46.754197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.762700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:46.766480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.770127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414b7e6-6fce-43e9-8a93-9726c8447652 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.780299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1081/2145] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.782745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.783217 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.783329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.783601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.783948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] PATH_INFO: `/v3/projects/47aa1e9c32034f62b9c720cc48ec7338/users/96c7129b47e94199bd8975bd3d3c98c7/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.786947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] RBAC: Authorizing `identity:create_grant(project_id=47aa1e9c32034f62b9c720cc48ec7338, user_id=96c7129b47e94199bd8975bd3d3c98c7, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.791204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.802307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f0d40a-fb44-437f-a63c-252971eab057 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.807997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1065/2146] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:46 2022] PUT /identity/v3/projects/47aa1e9c32034f62b9c720cc48ec7338/users/96c7129b47e94199bd8975bd3d3c98c7/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:46.817868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.893894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.894765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.895046 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.895315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] PATH_INFO: `/v3/projects/d00490854fcc4f75be1c3db02c7c3efd/users/f8b1282177bd457e9ba60f1d7c7d32da/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.915196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d00490854fcc4f75be1c3db02c7c3efd, user_id=f8b1282177bd457e9ba60f1d7c7d32da, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.918291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.941762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1082/2147] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:46 2022] PUT /identity/v3/projects/d00490854fcc4f75be1c3db02c7c3efd/users/f8b1282177bd457e9ba60f1d7c7d32da/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:46.945981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.969400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bab5a-78f7-438d-94d8-93e49ebe42be admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.048343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.049847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.050650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.052037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.052999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:47.053693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.056361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.069739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee563aef-edff-400e-8e3c-27d0bc91e6c3 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.075975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.077029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.077388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.077845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.078701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.080710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.090678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1066/2148] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:47.096093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.097727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1083/2149] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:47.098714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.109382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aba284-9825-48a9-865e-a432205c3344 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.113267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1b84f7-3942-4eb7-950c-71c8ca298b24 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.116947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.117658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.117940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.118382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.119033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.121135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.122030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aa89aa-cfbe-4457-b738-e7a6d5f706e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.123509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aa89aa-cfbe-4457-b738-e7a6d5f706e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.123793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aa89aa-cfbe-4457-b738-e7a6d5f706e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.124549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aa89aa-cfbe-4457-b738-e7a6d5f706e1 admin admin] PATH_INFO: `/v3/projects/d00490854fcc4f75be1c3db02c7c3efd/users/f8b1282177bd457e9ba60f1d7c7d32da/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.130770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aa89aa-cfbe-4457-b738-e7a6d5f706e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d00490854fcc4f75be1c3db02c7c3efd, user_id=f8b1282177bd457e9ba60f1d7c7d32da, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.139276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aa89aa-cfbe-4457-b738-e7a6d5f706e1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.166834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1084/2150] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:47 2022] PUT /identity/v3/projects/d00490854fcc4f75be1c3db02c7c3efd/users/f8b1282177bd457e9ba60f1d7c7d32da/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:47.169324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.169854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.170257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.190677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1067/2151] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/projects => generated 393 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:47.191933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.209836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] MFA Rules not processed for user `f8b1282177bd457e9ba60f1d7c7d32da`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:47.214436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.225098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:47.293256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ff36b7-84cf-49fe-b707-5e5cdd4d8769 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.298301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.298878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.299208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.300243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.300243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.301256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:47.302199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.326201 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1068/2152] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/users => generated 297 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:47.330694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.350583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac10386-92cc-4425-8073-142500a27bc3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.361961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.362784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.363102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.366987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.367101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1085/2153] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:47.367299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.367299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.382802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.384418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.387135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1069/2154] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:47.389849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.400638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa01660a-8649-4e39-a4a9-213141fc321f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.405806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.406416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.406694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.406964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.407608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.411775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.425263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1070/2155] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:47.432209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.451639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09508e0d-0a3c-4c0f-bc01-954ea1ea93c2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.463640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b952e335-c190-4fb2-88ee-71dc44343a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.464477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b952e335-c190-4fb2-88ee-71dc44343a60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.464766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b952e335-c190-4fb2-88ee-71dc44343a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.465070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b952e335-c190-4fb2-88ee-71dc44343a60 admin admin] PATH_INFO: `/v3/projects/0d54a2399cd14b849100cbd64f11cca3/users/c6bf23fb424f4567921cebc06b508d81/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.485123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b952e335-c190-4fb2-88ee-71dc44343a60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d54a2399cd14b849100cbd64f11cca3, user_id=c6bf23fb424f4567921cebc06b508d81, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.489494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b952e335-c190-4fb2-88ee-71dc44343a60 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.528716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1071/2156] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:47 2022] PUT /identity/v3/projects/0d54a2399cd14b849100cbd64f11cca3/users/c6bf23fb424f4567921cebc06b508d81/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:47.536600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.537038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.537321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.566197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ae048-1850-4d23-89bb-0ecc43221422 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.613752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] MFA Rules not processed for user `c6bf23fb424f4567921cebc06b508d81`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:47.616141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.720594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.721420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.722348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.723205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.794103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1072/2157] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:47.795659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.903448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:47.921995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.929111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.945709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b5e707-eac0-4413-aa11-f1b29d08158b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.958906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.958906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.958906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.958906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.959351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1086/2158] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:47 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:47.959444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.961230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.021401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1073/2159] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/projects => generated 393 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:48.026464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.041317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff0d68e-2968-4bc8-8f5b-7c0c2b172cd1 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.112688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.128428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:48.163672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.164448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.164830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.165289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.166028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.168603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.186735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0249e71-ab8a-4615-9ee0-bff3589ae4d3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.194104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.194922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.195208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.195648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.197773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.201446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1087/2160] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:48 2022] POST /identity/v3/users => generated 303 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:48.205537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.219428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da6bd5-bc8e-4c75-9f57-e99f1ba05a97 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.226322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.226934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.227213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.227484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.228134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.229786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.244825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1088/2161] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:48.245913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.259297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7668221a-cde3-4f9c-a2d6-b6de26373b01 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.265911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.266119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.266372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.266657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] PATH_INFO: `/v3/projects/22ebb7771eea4dc7afd16be14ab33a3a/users/4282754839604618b0418ff0aca105a4/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.281797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.284211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.287018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22ebb7771eea4dc7afd16be14ab33a3a, user_id=4282754839604618b0418ff0aca105a4, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.290544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.305989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1074/2162] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:48 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:48.312443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1089/2163] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:48 2022] PUT /identity/v3/projects/22ebb7771eea4dc7afd16be14ab33a3a/users/4282754839604618b0418ff0aca105a4/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:48.316674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.332251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06aaf52-73cb-4607-ad98-f31f4622bffe service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.410774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0559a0-b16c-40a8-bd5e-1e0892e483bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.411401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0559a0-b16c-40a8-bd5e-1e0892e483bb admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.411683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0559a0-b16c-40a8-bd5e-1e0892e483bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.411962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0559a0-b16c-40a8-bd5e-1e0892e483bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.412788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0559a0-b16c-40a8-bd5e-1e0892e483bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.415220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0559a0-b16c-40a8-bd5e-1e0892e483bb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.433148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1075/2164] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:48.437147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.455922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e12521-b177-4f75-9500-62390350a1b2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.471633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.472406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.473401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.474298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] PATH_INFO: `/v3/projects/22ebb7771eea4dc7afd16be14ab33a3a/users/4282754839604618b0418ff0aca105a4/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.479678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] RBAC: Authorizing `identity:create_grant(project_id=22ebb7771eea4dc7afd16be14ab33a3a, user_id=4282754839604618b0418ff0aca105a4, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.487642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.506934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1090/2165] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:48 2022] PUT /identity/v3/projects/22ebb7771eea4dc7afd16be14ab33a3a/users/4282754839604618b0418ff0aca105a4/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:48.518175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.518466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.518739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.595022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] MFA Rules not processed for user `4282754839604618b0418ff0aca105a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:48.596848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.613835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:48.730603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1076/2166] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:48 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:48.740617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.742674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.900420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1928755-57b2-4719-ad68-4a32261f205e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.908709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.909358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.912966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.913205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.951876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.954284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.992070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1091/2167] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:48 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:49.124925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:49.137661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f4c5b-486c-40a2-8c2b-1d8a6967e450 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:49.148303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c3e8de-1cc0-4827-b5e1-d4b9f1b480bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:49.149331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c3e8de-1cc0-4827-b5e1-d4b9f1b480bf service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:49.149451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c3e8de-1cc0-4827-b5e1-d4b9f1b480bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:49.149764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c3e8de-1cc0-4827-b5e1-d4b9f1b480bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:49.302106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c3e8de-1cc0-4827-b5e1-d4b9f1b480bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:49.303631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c3e8de-1cc0-4827-b5e1-d4b9f1b480bf service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:49.324304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1077/2168] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:49 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:50.354547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:50.356775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:50.545268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee9ae1e-bba7-4c00-8416-d7839e6d88e7 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:50.560536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:50.561457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:50.561884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:50.562302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] PATH_INFO: `/v3/users/a5a6fefa82e64e64a7e93d8295ceddc8` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:50.566185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5a6fefa82e64e64a7e93d8295ceddc8)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:50.568638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:50.639366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:50.718126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5a6fefa82e64e64a7e93d8295ceddc8'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:50.751951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5a6fefa82e64e64a7e93d8295ceddc8'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:50.761540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5a6fefa82e64e64a7e93d8295ceddc8'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:50.762442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5a6fefa82e64e64a7e93d8295ceddc8'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:50.763059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5a6fefa82e64e64a7e93d8295ceddc8'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:50.799551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:50.799551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:50.799551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:50.801030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1092/2169] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:50 2022] DELETE /identity/v3/users/a5a6fefa82e64e64a7e93d8295ceddc8 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:50.866683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:50.868638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:50.997734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1078/2170] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:51.008545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:51.010446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:51.162867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be12061e-55a5-4468-909c-35bcf2efd3d9 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:51.184527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96c9d01-d980-4dcf-9753-0cb766dbf484 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:51.185970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96c9d01-d980-4dcf-9753-0cb766dbf484 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:51.188219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96c9d01-d980-4dcf-9753-0cb766dbf484 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:51.188668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96c9d01-d980-4dcf-9753-0cb766dbf484 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:51.219020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96c9d01-d980-4dcf-9753-0cb766dbf484 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:51.221055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96c9d01-d980-4dcf-9753-0cb766dbf484 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:51.263368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1093/2171] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:51.702897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:51.706462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:51.739976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55c90cc-343d-4fd1-a6e3-77c8084cedb2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:51.752141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:51.754373 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:51.754665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:51.754939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] PATH_INFO: `/v3/users/a2ce06d6850f48f581d61be4b7377c26` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:51.756223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2ce06d6850f48f581d61be4b7377c26)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:51.761535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:51.851538 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:51.866045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2ce06d6850f48f581d61be4b7377c26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:51.879857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2ce06d6850f48f581d61be4b7377c26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:51.881251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2ce06d6850f48f581d61be4b7377c26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:51.882075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2ce06d6850f48f581d61be4b7377c26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:51.892994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2ce06d6850f48f581d61be4b7377c26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:51.928921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1079/2172] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:51 2022] DELETE /identity/v3/users/a2ce06d6850f48f581d61be4b7377c26 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:51.930040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f4dc31-4fd5-4394-bba9-954f87b85fd0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:51.930513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f4dc31-4fd5-4394-bba9-954f87b85fd0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:51.930738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f4dc31-4fd5-4394-bba9-954f87b85fd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:51.966998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-44f4dc31-4fd5-4394-bba9-954f87b85fd0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:51.968231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-44f4dc31-4fd5-4394-bba9-954f87b85fd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:52.096103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1094/2173] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:52.111881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:52.113400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:52.239671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7cfc89-851d-4b31-9ebc-456feaa1656e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:52.246062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cceb627-f6c8-453e-8998-6a15f238c8e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:52.246912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cceb627-f6c8-453e-8998-6a15f238c8e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:52.247372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cceb627-f6c8-453e-8998-6a15f238c8e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:52.247916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cceb627-f6c8-453e-8998-6a15f238c8e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:52.277083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cceb627-f6c8-453e-8998-6a15f238c8e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:52.279633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cceb627-f6c8-453e-8998-6a15f238c8e9 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:52.298522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1080/2174] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:52.612359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d0c48-7321-4a0c-838e-6fa86579dd93 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:52.612511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d0c48-7321-4a0c-838e-6fa86579dd93 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:52.614455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d0c48-7321-4a0c-838e-6fa86579dd93 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:52.622619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1095/2175] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:12:52 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:52.628740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:52.629062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:52.629335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:52.676590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:52.677876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:52.831304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1081/2176] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:12:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:52.855959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d0c48-7321-4a0c-838e-6fa86579dd93 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:52.857900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b02d0c48-7321-4a0c-838e-6fa86579dd93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:52.894516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d0c48-7321-4a0c-838e-6fa86579dd93 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:52.899836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:52.900538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:52.900824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:52.901150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] PATH_INFO: `/v3/users/2ab3ddf040e148789b3c67fc23d2da3d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:52.904383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ab3ddf040e148789b3c67fc23d2da3d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:52.908942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:52.935236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:52.938928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:52.980608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49e7b06-b68c-4170-ac37-865abde9656a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:52.997580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:52.998778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:52.999019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:53.000763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.008809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.025712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:53.028753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.041531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ab3ddf040e148789b3c67fc23d2da3d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.056148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1082/2177] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:53.083050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ab3ddf040e148789b3c67fc23d2da3d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.104376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ab3ddf040e148789b3c67fc23d2da3d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.106317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ab3ddf040e148789b3c67fc23d2da3d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.107138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ab3ddf040e148789b3c67fc23d2da3d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.169074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1096/2178] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:52 2022] DELETE /identity/v3/users/2ab3ddf040e148789b3c67fc23d2da3d => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:53.228235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:53.234010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.281455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1664236-a7d6-4554-b183-fe4027eaaf01 service placement] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:53.365557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:53.384685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:53.386191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.386676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.387255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] PATH_INFO: `/v3/projects/ce850c7c7ff34635b95a5caea9b83baa` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.388712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce850c7c7ff34635b95a5caea9b83baa)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:53.391724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.473174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce850c7c7ff34635b95a5caea9b83baa', 'request_id': 'req-dd242ac2-2478-44ed-8d96-30e6988bd49b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.497673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce850c7c7ff34635b95a5caea9b83baa', 'request_id': 'req-dd242ac2-2478-44ed-8d96-30e6988bd49b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.512093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:53.514035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.518315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce850c7c7ff34635b95a5caea9b83baa', 'request_id': 'req-dd242ac2-2478-44ed-8d96-30e6988bd49b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.535095 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c6013e-f5b1-4452-8b3d-61e961bed56d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:53.535914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] The token cache is being invalidate because project ce850c7c7ff34635b95a5caea9b83baa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:53.537411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.538581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.555766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1083/2179] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:53 2022] DELETE /identity/v3/projects/ce850c7c7ff34635b95a5caea9b83baa => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:53.635928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:53.647493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.750281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:53.750953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.751100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.753396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] PATH_INFO: `/v3/projects/dbda18c4ee4e4c9eb64f37e71bd862cd` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.755952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbda18c4ee4e4c9eb64f37e71bd862cd)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:53.761564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.826956 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd242ac2-2478-44ed-8d96-30e6988bd49b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:53.874242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbda18c4ee4e4c9eb64f37e71bd862cd', 'request_id': 'req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.885559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbda18c4ee4e4c9eb64f37e71bd862cd', 'request_id': 'req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.893362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbda18c4ee4e4c9eb64f37e71bd862cd', 'request_id': 'req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.916833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] The token cache is being invalidate because project dbda18c4ee4e4c9eb64f37e71bd862cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:53.917328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.917798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e22628b-f6dd-4136-a5c8-9c5ea2fc9aba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.919392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1097/2180] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:53 2022] DELETE /identity/v3/projects/dbda18c4ee4e4c9eb64f37e71bd862cd => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:53.924263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf435a6d-52e8-4d28-a686-3fa162cc7569 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.924544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf435a6d-52e8-4d28-a686-3fa162cc7569 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.924820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf435a6d-52e8-4d28-a686-3fa162cc7569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.938501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:53.940484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.940599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.940984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] PATH_INFO: `/v3/projects/3eab3c9cf27349c5ab214851e53938c1` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.943530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3eab3c9cf27349c5ab214851e53938c1)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:53.947126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.961733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cf435a6d-52e8-4d28-a686-3fa162cc7569 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:53.964564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cf435a6d-52e8-4d28-a686-3fa162cc7569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.985931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1098/2181] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:53.993139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:53.994793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.995247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.995705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.027375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3eab3c9cf27349c5ab214851e53938c1', 'request_id': 'req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.033364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:54.034592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.042657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3eab3c9cf27349c5ab214851e53938c1', 'request_id': 'req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.057880 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3eab3c9cf27349c5ab214851e53938c1', 'request_id': 'req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.076902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] The token cache is being invalidate because project 3eab3c9cf27349c5ab214851e53938c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:54.076902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.077870 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.079247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:54.085425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1084/2182] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:53 2022] DELETE /identity/v3/projects/3eab3c9cf27349c5ab214851e53938c1 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:54.086669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.089374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.119587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1099/2183] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:54.120974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.122928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.138173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454e706-e8a6-4c66-a3ac-d1d93e513967 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.146919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.147602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.147990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.148617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] PATH_INFO: `/v3/users/f8b1282177bd457e9ba60f1d7c7d32da` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.149884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8b1282177bd457e9ba60f1d7c7d32da)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.152773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.192998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4096b91-be78-4d2d-9b5f-b6aa7cb24774 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.202481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.203202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.203621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.204508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] PATH_INFO: `/v3/users/c6bf23fb424f4567921cebc06b508d81` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.206061 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6bf23fb424f4567921cebc06b508d81)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.208349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.256412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8b1282177bd457e9ba60f1d7c7d32da'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.295413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8b1282177bd457e9ba60f1d7c7d32da'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.297087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6bf23fb424f4567921cebc06b508d81'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.306864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8b1282177bd457e9ba60f1d7c7d32da'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.307957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8b1282177bd457e9ba60f1d7c7d32da'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.308595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8b1282177bd457e9ba60f1d7c7d32da'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.314227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6bf23fb424f4567921cebc06b508d81'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.315552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6bf23fb424f4567921cebc06b508d81'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.318645 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6bf23fb424f4567921cebc06b508d81'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.331573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1100/2184] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:54 2022] DELETE /identity/v3/users/f8b1282177bd457e9ba60f1d7c7d32da => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:54.332782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.335639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.339185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ffbe5924-4fcb-4fad-a1ff-e5c819390cf4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6bf23fb424f4567921cebc06b508d81'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.384104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1085/2185] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:54 2022] DELETE /identity/v3/users/c6bf23fb424f4567921cebc06b508d81 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:54.388788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b7311-e66c-44b0-b1b9-f6fbc807c25a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.388788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b7311-e66c-44b0-b1b9-f6fbc807c25a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.388788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b7311-e66c-44b0-b1b9-f6fbc807c25a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.443023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-220b7311-e66c-44b0-b1b9-f6fbc807c25a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:54.446604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-220b7311-e66c-44b0-b1b9-f6fbc807c25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.464745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe004466-eb44-4b27-a1c4-e1b4c7c1ed7f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.518114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:54.616942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1086/2186] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:54.618575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.619476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.620476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.630884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.631516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.631795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.632061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.632715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:54.633111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.651546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.654280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:54.655029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] MFA Rules not processed for user `96c7129b47e94199bd8975bd3d3c98c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:54.656507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.690771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1101/2187] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:54.692243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.694312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.761337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f360428-0100-4c24-a6e5-f4fe77c06495 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.766941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.767747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1087/2188] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:54 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:54.768510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.768980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.769413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.773846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.782003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.785898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.787760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.797554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8562a90b-734e-46e5-bba5-23333bdb4a20 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.802305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1102/2189] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:54.803174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.803291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.804247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.804542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.804663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.805002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.805711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.807901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.848936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1088/2190] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:54 2022] POST /identity/v3/projects => generated 369 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:54.853250 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.903392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2869192-0bf0-40fc-bdcc-5c6a72d977f8 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.964841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f4f4e-4a0a-4040-b755-313c0a3367aa admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.970710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.971328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.971602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.971875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.972511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.984906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.004657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.005458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.005857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.006179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.009109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.025567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1089/2191] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:54 2022] POST /identity/v3/users => generated 285 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:55.029190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.046622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7a08c0-027c-495f-975f-114a8aba7eb6 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.055588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.057123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.057554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.058156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.059343 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.062951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:55.066573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.095026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1090/2192] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:55 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:55.096303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.101537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.124565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91917db0-773c-4526-8c84-e02fe465373e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.130977 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.131555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.132005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.132283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] PATH_INFO: `/v3/projects/d00490854fcc4f75be1c3db02c7c3efd` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.133511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=d00490854fcc4f75be1c3db02c7c3efd)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.135350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.145703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.147606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.153243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:55.177171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1103/2193] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:55.185756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.212978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa2d24f-8d23-420d-b908-c8d683373fc9 service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.224562 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.232284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.232681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.235002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.235751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.240285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.254761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1104/2194] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:55 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:55.263612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.275016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd00490854fcc4f75be1c3db02c7c3efd', 'request_id': 'req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.298678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f7ec0f-26d6-4df6-99f6-c6efcb1e7119 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.301711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd00490854fcc4f75be1c3db02c7c3efd', 'request_id': 'req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.313215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd00490854fcc4f75be1c3db02c7c3efd', 'request_id': 'req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.325633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] The token cache is being invalidate because project d00490854fcc4f75be1c3db02c7c3efd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:55.326125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.326612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.328912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1091/2195] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:55 2022] DELETE /identity/v3/projects/d00490854fcc4f75be1c3db02c7c3efd => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:55.329279 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.330740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.408318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8602ee55-aa2a-4dac-a64f-6d0aa8ac32a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.409015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602ee55-aa2a-4dac-a64f-6d0aa8ac32a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.409297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602ee55-aa2a-4dac-a64f-6d0aa8ac32a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.409568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602ee55-aa2a-4dac-a64f-6d0aa8ac32a5 admin admin] PATH_INFO: `/v3/projects/e5ea015e37d34bea92f0877063697d96/users/0ac4bc384d5341aa8fc19634a0d897a2/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.440648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8602ee55-aa2a-4dac-a64f-6d0aa8ac32a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5ea015e37d34bea92f0877063697d96, user_id=0ac4bc384d5341aa8fc19634a0d897a2, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.444862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8602ee55-aa2a-4dac-a64f-6d0aa8ac32a5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.471273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b03d335-43c8-4f83-8ab3-678b2cf2b0ea admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.471569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1105/2196] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:55 2022] PUT /identity/v3/projects/e5ea015e37d34bea92f0877063697d96/users/0ac4bc384d5341aa8fc19634a0d897a2/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:55.483346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a75fc7-19ba-4bd9-81fa-2ab477dfa23d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.483933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a75fc7-19ba-4bd9-81fa-2ab477dfa23d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.484218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a75fc7-19ba-4bd9-81fa-2ab477dfa23d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.514355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d5a75fc7-19ba-4bd9-81fa-2ab477dfa23d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:55.521914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d5a75fc7-19ba-4bd9-81fa-2ab477dfa23d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.551594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.552796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.554035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.555651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] PATH_INFO: `/v3/projects/0d54a2399cd14b849100cbd64f11cca3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.557334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d54a2399cd14b849100cbd64f11cca3)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.559777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.640593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1106/2197] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:55.641984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.642800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.643075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.674924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:55.680439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d54a2399cd14b849100cbd64f11cca3', 'request_id': 'req-188d545b-068c-4314-9c68-1d1884589e10'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.692900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d54a2399cd14b849100cbd64f11cca3', 'request_id': 'req-188d545b-068c-4314-9c68-1d1884589e10'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.705447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d54a2399cd14b849100cbd64f11cca3', 'request_id': 'req-188d545b-068c-4314-9c68-1d1884589e10'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.711249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] MFA Rules not processed for user `0ac4bc384d5341aa8fc19634a0d897a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:55.714107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.730827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] The token cache is being invalidate because project 0d54a2399cd14b849100cbd64f11cca3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:55.731119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.731853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-188d545b-068c-4314-9c68-1d1884589e10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.737814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1092/2198] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:55 2022] DELETE /identity/v3/projects/0d54a2399cd14b849100cbd64f11cca3 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:55.739863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.740160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.740439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.764016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:55.787702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:55.789723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.832444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1107/2199] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:55 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:55.833729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.836901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.915793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1093/2200] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:55.916685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:55.919964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:55.967238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8013b64f-ca90-431e-ba23-cdaf1d98a459 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.974411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.975288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.975763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.976214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.977105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.979220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.988796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ddee61-c277-4876-a976-ad27acd91d3b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.998494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.999176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.999462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.999734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] PATH_INFO: `/v3/users/4282754839604618b0418ff0aca105a4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.001055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4282754839604618b0418ff0aca105a4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.004227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.026038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1094/2201] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:55 2022] POST /identity/v3/projects => generated 369 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:56.027031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.089107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4282754839604618b0418ff0aca105a4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.108644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4282754839604618b0418ff0aca105a4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.114244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56bc739-6bbd-4c18-bd00-718d1a0242bd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.122470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4282754839604618b0418ff0aca105a4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.124066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4282754839604618b0418ff0aca105a4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.124998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4282754839604618b0418ff0aca105a4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.126189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.128090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.133306 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.133939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.135164 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:56.135834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.139062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.155094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1095/2202] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:56.156090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.158311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.162140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1108/2203] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:55 2022] DELETE /identity/v3/users/4282754839604618b0418ff0aca105a4 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:56.162140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.163109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.200730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14085416-2972-414c-a3b5-8e2ace196941 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.259333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:56.283557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:56.293390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4db631b-fbd5-4d3e-a618-6b4383dd1ec5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.306133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.307078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.307078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.307293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.307695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.307939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.308639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.309324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.310076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.310800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.312530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.313090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.352305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1096/2204] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:56 2022] POST /identity/v3/users => generated 291 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:56.354129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.361735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1109/2205] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:56 2022] POST /identity/v3/projects => generated 379 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:56.368750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.369521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe655dfa-ca89-4fc1-b495-bfce6ed2cce7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.457181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.458661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.459233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.459716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.460671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.463057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.472934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15950026-775e-455d-9a50-56a7722fd1d7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.481522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.482708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.483434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.483852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.484627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.489572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1097/2206] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:56.490681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.490815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.501960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0c8942-8d3a-4ad6-88a1-b24cc39375c1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.508344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.509116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.509503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.509818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] PATH_INFO: `/v3/projects/c22ff83356ad42ddbfaf79760a3a4a38/users/cd769ea9e99d42f7b96a895df49e91d8/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.531141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c22ff83356ad42ddbfaf79760a3a4a38, user_id=cd769ea9e99d42f7b96a895df49e91d8, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.534861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.538244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1110/2207] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:56 2022] POST /identity/v3/users => generated 290 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:12:56.543379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.555870 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0649f3e4-ba1d-4c8b-8419-1cc8af76c021 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.561522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1098/2208] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:56 2022] PUT /identity/v3/projects/c22ff83356ad42ddbfaf79760a3a4a38/users/cd769ea9e99d42f7b96a895df49e91d8/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:56.572240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.652208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.652833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.653145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.653425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.654106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.655786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.669229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1111/2209] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:56.670488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.672792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.691476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b25d94c-d11c-4450-a217-284dcf934b5f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.692088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce379d-825a-4138-8b11-2052b40e7af9 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.700638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.701412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.701903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.702237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.703011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.704741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.705428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.705708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.705988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] PATH_INFO: `/v3/projects/22ebb7771eea4dc7afd16be14ab33a3a` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.707762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] RBAC: Authorizing `identity:delete_project(project_id=22ebb7771eea4dc7afd16be14ab33a3a)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.708020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.709394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.721111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1099/2210] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:56.722068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.733534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceef8b66-3695-4f57-8f9a-df1082e2dc56 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.740401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.741403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.741701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.741972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.742682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.744283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.761123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1100/2211] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:56.761992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.775230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8541e6-03a3-4838-ba68-a60d67222589 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.793558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22ebb7771eea4dc7afd16be14ab33a3a', 'request_id': 'req-ffb8a518-d8a7-452a-8f59-37c80e1afffb'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.806488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22ebb7771eea4dc7afd16be14ab33a3a', 'request_id': 'req-ffb8a518-d8a7-452a-8f59-37c80e1afffb'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.809584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:56.816161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22ebb7771eea4dc7afd16be14ab33a3a', 'request_id': 'req-ffb8a518-d8a7-452a-8f59-37c80e1afffb'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.824883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:56.842640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] The token cache is being invalidate because project 22ebb7771eea4dc7afd16be14ab33a3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:56.843033 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.843713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:56.845384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1112/2212] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:56 2022] DELETE /identity/v3/projects/22ebb7771eea4dc7afd16be14ab33a3a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:56.846162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.847498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.866791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d4942c-2c96-4be6-8cd1-e89c88d9e958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.867803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d4942c-2c96-4be6-8cd1-e89c88d9e958 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.868239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d4942c-2c96-4be6-8cd1-e89c88d9e958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.868738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d4942c-2c96-4be6-8cd1-e89c88d9e958 admin admin] PATH_INFO: `/v3/projects/c22ff83356ad42ddbfaf79760a3a4a38/users/cd769ea9e99d42f7b96a895df49e91d8/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.871597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d4942c-2c96-4be6-8cd1-e89c88d9e958 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c22ff83356ad42ddbfaf79760a3a4a38, user_id=cd769ea9e99d42f7b96a895df49e91d8, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:56.875129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d4942c-2c96-4be6-8cd1-e89c88d9e958 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:56.917832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1101/2213] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:56 2022] PUT /identity/v3/projects/c22ff83356ad42ddbfaf79760a3a4a38/users/cd769ea9e99d42f7b96a895df49e91d8/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:56.917832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a659d1-7f98-4414-8c0c-bcfa01128293 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.917832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a659d1-7f98-4414-8c0c-bcfa01128293 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.917832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a659d1-7f98-4414-8c0c-bcfa01128293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.942659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8a518-d8a7-452a-8f59-37c80e1afffb admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.946619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a5a659d1-7f98-4414-8c0c-bcfa01128293 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:56.947998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a5a659d1-7f98-4414-8c0c-bcfa01128293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.028500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971c5025-db18-43d5-80b4-b9c998da966c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.029209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971c5025-db18-43d5-80b4-b9c998da966c admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.029489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971c5025-db18-43d5-80b4-b9c998da966c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.029758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971c5025-db18-43d5-80b4-b9c998da966c admin admin] PATH_INFO: `/v3/projects/a56795d7c1df4a4c9a9831cb59a4cb7d/users/79bcb9db31e24b6981a3e356debca53b/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.032131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1102/2214] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:57.033680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.033999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.034277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.056614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971c5025-db18-43d5-80b4-b9c998da966c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a56795d7c1df4a4c9a9831cb59a4cb7d, user_id=79bcb9db31e24b6981a3e356debca53b, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.060071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971c5025-db18-43d5-80b4-b9c998da966c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.075725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] MFA Rules not processed for user `cd769ea9e99d42f7b96a895df49e91d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:57.077699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.086185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1113/2215] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:56 2022] PUT /identity/v3/projects/a56795d7c1df4a4c9a9831cb59a4cb7d/users/79bcb9db31e24b6981a3e356debca53b/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:57.087460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9b34d-5886-44d7-bc92-9ee1e9206873 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.088026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9b34d-5886-44d7-bc92-9ee1e9206873 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.088253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b9b34d-5886-44d7-bc92-9ee1e9206873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.109510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d4b9b34d-5886-44d7-bc92-9ee1e9206873 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:57.110772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d4b9b34d-5886-44d7-bc92-9ee1e9206873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.194943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1114/2216] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:57.196164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.196453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.196729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.199518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1103/2217] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:57.200525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.214682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f72cec-b2f2-4c92-a858-8e89ad6eb34a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.219811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.220564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.220736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.221041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.221687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:57.221991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.223690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.238219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1104/2218] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:57.239041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.242376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] MFA Rules not processed for user `79bcb9db31e24b6981a3e356debca53b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:57.245799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.246393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.257633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:57.319019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:57.338239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c99297-f142-48a4-8de2-b18a548f72d0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.342973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.343900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.344411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.345762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.359935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.361708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.375120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1115/2219] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:57.376186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.382008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1105/2220] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:57.382854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.387659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d226d0c5-4721-4f4f-a2f6-2f4833ea6e84 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.393350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.394041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.394320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.394597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.395228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.397187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.401973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42db60d-18d6-4c09-92bf-d4ed0c990710 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.416959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.417858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.419247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.419247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.419815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.422552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.447133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1116/2221] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/projects => generated 367 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:57.451448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.491216 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1106/2222] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/projects => generated 379 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:57.492196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.557716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46ac5c8-df42-4064-a571-3d11302cce73 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.573470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626e03b-3261-486d-996d-92df9c8c6127 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.577863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.578603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.578960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.579517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.584946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.654946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.655542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.655817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.656083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.656719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.659228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.701983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.706315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.711279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1117/2223] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/users => generated 284 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:57.711279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.724177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc5dcd-730c-4d34-be52-6ab4e43932cc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.729328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1107/2224] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:57.730692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.731877 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.732411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.732778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.733553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.733695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.735883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.749836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f0cc15-9aea-47ca-af17-54fe7e9c31e9 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.755297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.756005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.756280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.756689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.757534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.759340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.766117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1118/2225] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/users => generated 290 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:57.767782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.772155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1108/2226] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:57.775785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.787099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c4218-ed4d-48ac-b4f6-c9aa3564ab76 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.788657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:57.793899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.795007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.795007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.795151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.795721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.797496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.799318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bcdc5c-cfa0-47d9-b80d-a0bbe8abc250 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.812955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1109/2227] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:57.813929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.815689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.815971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.816356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.816898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.818672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.821040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.840503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f3e9bd-0f11-4dae-921f-77fb9d1c36a0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.845535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1119/2228] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:57.849244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.855583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.856272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.856669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.856828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] PATH_INFO: `/v3/projects/1c9a601330a9479e89ee25506a57aeff/users/393f6e96e0c74640a9485147e305786f/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.871007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ee3639-cae2-46a2-b0ed-997a80ae460a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.884210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11eaf8-fd49-4b7f-95e8-b9aa4a6251dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.884898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11eaf8-fd49-4b7f-95e8-b9aa4a6251dd admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.885168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11eaf8-fd49-4b7f-95e8-b9aa4a6251dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.885444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11eaf8-fd49-4b7f-95e8-b9aa4a6251dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.886183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b11eaf8-fd49-4b7f-95e8-b9aa4a6251dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.887704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b11eaf8-fd49-4b7f-95e8-b9aa4a6251dd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.887947 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c9a601330a9479e89ee25506a57aeff, user_id=393f6e96e0c74640a9485147e305786f, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.891163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.910040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1110/2229] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:57 2022] PUT /identity/v3/projects/1c9a601330a9479e89ee25506a57aeff/users/393f6e96e0c74640a9485147e305786f/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:57.917942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1120/2230] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:57 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:57.920500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.920706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.921008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.921371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.961121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] MFA Rules not processed for user `393f6e96e0c74640a9485147e305786f`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:57.963626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.979068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:58.021065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6743b46-3abd-441e-b4b8-ed79a405150e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.027484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1893207-d7ec-4e63-8169-4fe9eb9d982a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.028272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1893207-d7ec-4e63-8169-4fe9eb9d982a admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.028623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1893207-d7ec-4e63-8169-4fe9eb9d982a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.028990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1893207-d7ec-4e63-8169-4fe9eb9d982a admin admin] PATH_INFO: `/v3/projects/1ffc6f4d45494f6294514a9079188f8f/users/e80bccd2a0a54a1dbdfec6b3da368e1b/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.047514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1893207-d7ec-4e63-8169-4fe9eb9d982a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ffc6f4d45494f6294514a9079188f8f, user_id=e80bccd2a0a54a1dbdfec6b3da368e1b, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.050687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1893207-d7ec-4e63-8169-4fe9eb9d982a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.070867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1111/2231] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:57 2022] PUT /identity/v3/projects/1ffc6f4d45494f6294514a9079188f8f/users/e80bccd2a0a54a1dbdfec6b3da368e1b/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:58.081047 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.081507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.081909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.097108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1121/2232] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:58.103698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.129159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] MFA Rules not processed for user `e80bccd2a0a54a1dbdfec6b3da368e1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:58.130520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:58.136242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e6c20b-756a-473f-90e6-fb98da464228 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.250872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.251468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.251754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.252187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.252841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.256405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.266620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1112/2233] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:58.271588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.287451 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af76ab66-8ff7-490e-a4ab-388b4b8b33d4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.294699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.295424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.295828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.296274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.297635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.300476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.341975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1122/2234] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/projects => generated 367 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:58.355247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.360491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1113/2235] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/projects => generated 377 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:58.363929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.457678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0635ca5-4389-4838-aded-98ccdbd842f5 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.462833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.464394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.464394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.464394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.464828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.466969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.492536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba7aacc-cb43-4f52-85bf-613a7a0ecf2a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.494017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:58.501651 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.502540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.503019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.503495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.504240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.507190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.507509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1114/2236] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/users => generated 295 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:58.508534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.520552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db6d917-0558-4f93-9eff-c02d85d5aab7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.528349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.530448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.530683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.530969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.531905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.533799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.536909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:58.538053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1123/2237] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/users => generated 290 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:58.543329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.557293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df543962-2499-451d-af99-f1a82edbfe6c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.560784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1115/2238] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:58.561859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.563671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.564438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.565069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.565415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.571560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.571560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.574000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57970e62-d758-4b5c-b0b4-79f2fa688698 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.578669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.579344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.586470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.586470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] PATH_INFO: `/v3/projects/046f78fca72548068f4bb604b30e3238/users/1e9b25a2ed6447e5936b025ff390fe42/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.597755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1124/2239] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:58.601920 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.602390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=046f78fca72548068f4bb604b30e3238, user_id=1e9b25a2ed6447e5936b025ff390fe42, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.607706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.614968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0370f2f-525b-4ce9-ae72-4e37564c63a3 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.631086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.631538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.631822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.632094 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] PATH_INFO: `/v3/projects/b24151f6abf849a38789a40cc923e561/users/27684af6625944da98274dc3277c9e38/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.635448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1116/2240] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:58 2022] PUT /identity/v3/projects/046f78fca72548068f4bb604b30e3238/users/1e9b25a2ed6447e5936b025ff390fe42/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:58.636479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.653199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b24151f6abf849a38789a40cc923e561, user_id=27684af6625944da98274dc3277c9e38, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.657050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.695637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1125/2241] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:58 2022] PUT /identity/v3/projects/b24151f6abf849a38789a40cc923e561/users/27684af6625944da98274dc3277c9e38/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:58.699833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.747771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3cc297-1dcb-413d-a008-73c9baba2c5b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.781633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742d5ca9-91ba-4000-b69e-fcff47a74e93 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.790179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.791430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.792255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.792755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.794478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.796703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.810911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1126/2242] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:58.820391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.847067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5df2af-b372-4f88-ba22-c1fd5b3aff4e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.852267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7fde31-f4e8-415f-99e3-314993f4b577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.852963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7fde31-f4e8-415f-99e3-314993f4b577 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.853244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7fde31-f4e8-415f-99e3-314993f4b577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.853516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7fde31-f4e8-415f-99e3-314993f4b577 admin admin] PATH_INFO: `/v3/projects/b24151f6abf849a38789a40cc923e561/users/27684af6625944da98274dc3277c9e38/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.855493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7fde31-f4e8-415f-99e3-314993f4b577 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b24151f6abf849a38789a40cc923e561, user_id=27684af6625944da98274dc3277c9e38, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.857768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.858633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.860397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7fde31-f4e8-415f-99e3-314993f4b577 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.869485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.870059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.871936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.873075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1127/2243] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:58 2022] PUT /identity/v3/projects/b24151f6abf849a38789a40cc923e561/users/27684af6625944da98274dc3277c9e38/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:58.874207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.886477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.886782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.887047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.890747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1117/2244] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:58.895754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.910737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a70c05-7846-41e6-8778-47ab28ffe1b1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.933312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] MFA Rules not processed for user `27684af6625944da98274dc3277c9e38`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:58.933312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:59.010658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990549b3-c40a-47f5-83d6-0587b3d6a774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.011573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990549b3-c40a-47f5-83d6-0587b3d6a774 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.011860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990549b3-c40a-47f5-83d6-0587b3d6a774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.012141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990549b3-c40a-47f5-83d6-0587b3d6a774 admin admin] PATH_INFO: `/v3/projects/046f78fca72548068f4bb604b30e3238/users/1e9b25a2ed6447e5936b025ff390fe42/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.014147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990549b3-c40a-47f5-83d6-0587b3d6a774 admin admin] RBAC: Authorizing `identity:create_grant(project_id=046f78fca72548068f4bb604b30e3238, user_id=1e9b25a2ed6447e5936b025ff390fe42, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.016883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990549b3-c40a-47f5-83d6-0587b3d6a774 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.036837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1128/2245] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:59.044925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1118/2246] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:58 2022] PUT /identity/v3/projects/046f78fca72548068f4bb604b30e3238/users/1e9b25a2ed6447e5936b025ff390fe42/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:59.047745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.049874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:59.053748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.054234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.054583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.074530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:59.083546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:59.094844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] MFA Rules not processed for user `1e9b25a2ed6447e5936b025ff390fe42`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:59.096600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:59.123236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cff6a52-6995-4760-b745-8908359030a2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.187571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.188245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.188483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.188723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.189297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1119/2247] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:12:59 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:59.200897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.222628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0349a041-3357-4358-ab57-9d35e042fe2e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.227472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.228272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.228623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.228998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.241478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.243190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.268340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1120/2248] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:59 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:59.284023 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.285695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.314066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1129/2249] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:59 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:59.373041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.382365 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b45f8d7-7344-4fb8-b248-6aadfcaa9b51 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.390191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.390889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.391577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.391699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.485394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.489905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.491529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.496417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a87350-ecc1-4906-b2ee-4ebcdd581de3 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.500211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623888d9-0060-407a-abf8-081099022d7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.500967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623888d9-0060-407a-abf8-081099022d7b service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.501253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623888d9-0060-407a-abf8-081099022d7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.502090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623888d9-0060-407a-abf8-081099022d7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.515416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1121/2250] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:59 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:59.580256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623888d9-0060-407a-abf8-081099022d7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.581769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623888d9-0060-407a-abf8-081099022d7b service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.591203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1130/2251] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:59 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:02.093536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.095103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.162514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fc8aa-da87-4a6e-bbc2-ce3a7da1abab service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.166988 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.167650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.167934 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.168208 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] PATH_INFO: `/v3/users/0ac4bc384d5341aa8fc19634a0d897a2` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.169362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ac4bc384d5341aa8fc19634a0d897a2)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:02.171308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:02.194714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:02.228751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ac4bc384d5341aa8fc19634a0d897a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.237151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ac4bc384d5341aa8fc19634a0d897a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.237886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ac4bc384d5341aa8fc19634a0d897a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.238826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ac4bc384d5341aa8fc19634a0d897a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.251377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ac4bc384d5341aa8fc19634a0d897a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.270251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1122/2252] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:02 2022] DELETE /identity/v3/users/0ac4bc384d5341aa8fc19634a0d897a2 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:02.274878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.275606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.275740 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.295449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:02.296815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.321914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:02.368813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1131/2253] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:02.381914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.383406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.445007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aca34e-c3ed-40aa-a78d-e7a5cf7522cd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.451399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.452032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.452318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.452631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.463000 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:02.464420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:02.473948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1123/2254] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:02.757573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.759245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.778847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ae68f-7d67-42e0-b392-2953978749aa None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.789844 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.790724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.790999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.791268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] PATH_INFO: `/v3/projects/e5ea015e37d34bea92f0877063697d96` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.792425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5ea015e37d34bea92f0877063697d96)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:02.794388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:02.853068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5ea015e37d34bea92f0877063697d96', 'request_id': 'req-c289fac0-3833-412b-8325-f696117776f5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.862307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5ea015e37d34bea92f0877063697d96', 'request_id': 'req-c289fac0-3833-412b-8325-f696117776f5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.870627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5ea015e37d34bea92f0877063697d96', 'request_id': 'req-c289fac0-3833-412b-8325-f696117776f5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.889224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] The token cache is being invalidate because project e5ea015e37d34bea92f0877063697d96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:02.889524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.890366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:02.893367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1132/2255] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:02 2022] DELETE /identity/v3/projects/e5ea015e37d34bea92f0877063697d96 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:02.897275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.909037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e81a1c-93be-4ae4-8da1-fb17715461bf service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.957368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.958817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.991611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.992577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.992854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.993157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] PATH_INFO: `/v3/users/cd769ea9e99d42f7b96a895df49e91d8` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.994292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd769ea9e99d42f7b96a895df49e91d8)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:02.997161 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.055529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c289fac0-3833-412b-8325-f696117776f5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:03.059849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:03.060896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:03.061839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.062156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd769ea9e99d42f7b96a895df49e91d8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.062278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.062497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] PATH_INFO: `/v3/users/79bcb9db31e24b6981a3e356debca53b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.063779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] RBAC: Authorizing `identity:delete_user(user_id=79bcb9db31e24b6981a3e356debca53b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:03.065980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.078589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd769ea9e99d42f7b96a895df49e91d8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.080109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd769ea9e99d42f7b96a895df49e91d8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.080825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd769ea9e99d42f7b96a895df49e91d8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.091327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e61e0a64-eb73-48ab-a721-800f030a54d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd769ea9e99d42f7b96a895df49e91d8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.117516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1124/2256] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:02 2022] DELETE /identity/v3/users/cd769ea9e99d42f7b96a895df49e91d8 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:03.134771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79bcb9db31e24b6981a3e356debca53b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.136776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:03.149394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79bcb9db31e24b6981a3e356debca53b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.158755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79bcb9db31e24b6981a3e356debca53b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.163069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79bcb9db31e24b6981a3e356debca53b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.163720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79bcb9db31e24b6981a3e356debca53b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.180097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1133/2257] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:02 2022] DELETE /identity/v3/users/79bcb9db31e24b6981a3e356debca53b => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:03.185313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.185612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.185886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.217199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:03.218498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.299137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1125/2258] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:03.309839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:03.311749 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.419894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ba3986-807e-4abd-a127-e00bd108995a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:03.424191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c8e5ba-9014-432e-b317-bb6589ea7bd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:03.424780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c8e5ba-9014-432e-b317-bb6589ea7bd5 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.425282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c8e5ba-9014-432e-b317-bb6589ea7bd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.425555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c8e5ba-9014-432e-b317-bb6589ea7bd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.439277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c8e5ba-9014-432e-b317-bb6589ea7bd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:03.442288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c8e5ba-9014-432e-b317-bb6589ea7bd5 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.457204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1134/2259] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:03.670546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:03.671956 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.691080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eb3f59-6f1d-432a-81fb-ef95e3d080e6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:03.696326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:03.697736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.697736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.697736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] PATH_INFO: `/v3/projects/c22ff83356ad42ddbfaf79760a3a4a38` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.698956 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] RBAC: Authorizing `identity:delete_project(project_id=c22ff83356ad42ddbfaf79760a3a4a38)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:03.700992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.778371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c22ff83356ad42ddbfaf79760a3a4a38', 'request_id': 'req-ae5659c5-68da-4d3b-97f7-94b543e614be'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.788607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c22ff83356ad42ddbfaf79760a3a4a38', 'request_id': 'req-ae5659c5-68da-4d3b-97f7-94b543e614be'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.796597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c22ff83356ad42ddbfaf79760a3a4a38', 'request_id': 'req-ae5659c5-68da-4d3b-97f7-94b543e614be'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.811085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] The token cache is being invalidate because project c22ff83356ad42ddbfaf79760a3a4a38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:03.811417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.812277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ae5659c5-68da-4d3b-97f7-94b543e614be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:03.814419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1126/2260] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:03 2022] DELETE /identity/v3/projects/c22ff83356ad42ddbfaf79760a3a4a38 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:03.823273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.825454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.826144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.849883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:03.851137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.921774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1135/2261] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:03.932159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.933221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.934071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.956788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:03.958489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.962249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:03.963854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.973178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5c1362-0983-454e-83bb-16177448ec29 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:03.978334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1127/2262] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:03.982218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:03.982529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:03.983181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.983521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.983793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] PATH_INFO: `/v3/projects/a56795d7c1df4a4c9a9831cb59a4cb7d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.984984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] RBAC: Authorizing `identity:delete_project(project_id=a56795d7c1df4a4c9a9831cb59a4cb7d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:03.988109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.998530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244f3b5-d63e-4c5d-a71f-4dcc709e4942 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.006359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.007135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.007535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.007966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.009071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:04.009668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.012448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.033105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1128/2263] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:04.035702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.047486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2850effc-c5c7-44d2-aa5f-3a641ecb9c4a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.049980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a56795d7c1df4a4c9a9831cb59a4cb7d', 'request_id': 'req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.060190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a56795d7c1df4a4c9a9831cb59a4cb7d', 'request_id': 'req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.068655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a56795d7c1df4a4c9a9831cb59a4cb7d', 'request_id': 'req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.070640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:04.080411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:04.087113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] The token cache is being invalidate because project a56795d7c1df4a4c9a9831cb59a4cb7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:04.087757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.089182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.090932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1136/2264] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:03 2022] DELETE /identity/v3/projects/a56795d7c1df4a4c9a9831cb59a4cb7d => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:04.094859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.096367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.123390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.124089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.124369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.124642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.125316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.127859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.162484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1129/2265] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:04 2022] POST /identity/v3/projects => generated 387 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:04.166652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.168267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.168619 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250c2bb7-c940-4c90-a5a5-3ed6fcc1fcef admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.231566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.232379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.232663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.232962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] PATH_INFO: `/v3/users/e80bccd2a0a54a1dbdfec6b3da368e1b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.234155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] RBAC: Authorizing `identity:delete_user(user_id=e80bccd2a0a54a1dbdfec6b3da368e1b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.236218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.237275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef798f9-b1c2-42ea-8ff7-a03f976fa1fb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.242309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.242483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.242759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.243029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.243699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.245646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.271899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1130/2266] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:04 2022] POST /identity/v3/users => generated 294 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:04.275333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.285235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc01b0-9eb4-42b1-b83e-6423d2b8d44d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.288268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e80bccd2a0a54a1dbdfec6b3da368e1b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.290657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.291523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.291806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.292081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.292994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.295049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.302739 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e80bccd2a0a54a1dbdfec6b3da368e1b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.309868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1131/2267] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:04.310017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e80bccd2a0a54a1dbdfec6b3da368e1b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.310511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e80bccd2a0a54a1dbdfec6b3da368e1b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.310861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.311131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e80bccd2a0a54a1dbdfec6b3da368e1b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.312537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.326165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1137/2268] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:04 2022] DELETE /identity/v3/users/e80bccd2a0a54a1dbdfec6b3da368e1b => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:04.326844 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604f32ec-0c21-4717-a8a5-e6fba2c7a015 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.389439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89047b92-d2e3-4115-891a-1838cbcf0a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.390600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89047b92-d2e3-4115-891a-1838cbcf0a31 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.390600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89047b92-d2e3-4115-891a-1838cbcf0a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.390784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89047b92-d2e3-4115-891a-1838cbcf0a31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.391376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89047b92-d2e3-4115-891a-1838cbcf0a31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.393788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89047b92-d2e3-4115-891a-1838cbcf0a31 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.405207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1132/2269] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:04.419936 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.430704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afda505-c9a0-4c5a-bf28-2bf4c73b6afd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.435804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.436468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.436745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.437052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] PATH_INFO: `/v3/projects/0b10a783b984421788f1a32e3270694e/users/1759cacb633948e2b705e8bd42aaa923/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.453175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b10a783b984421788f1a32e3270694e, user_id=1759cacb633948e2b705e8bd42aaa923, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.456602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.459359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:04.472291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1138/2270] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:04 2022] PUT /identity/v3/projects/0b10a783b984421788f1a32e3270694e/users/1759cacb633948e2b705e8bd42aaa923/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:04.484854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.485167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.485521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.518679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] MFA Rules not processed for user `1759cacb633948e2b705e8bd42aaa923`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:04.519855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.534052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:04.601659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1133/2271] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:04.606090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.641693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.643664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.679196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cda20f1-1e1c-45a4-9ea3-676633188c04 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.684101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.684708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.685048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.685309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.685951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.687860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.714883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e041cc7-101a-46e8-bafd-8b713efeeb06 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.719444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.720654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.720963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.721243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] PATH_INFO: `/v3/projects/1ffc6f4d45494f6294514a9079188f8f` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.721776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1139/2272] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:04 2022] POST /identity/v3/projects => generated 385 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:04.722684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ffc6f4d45494f6294514a9079188f8f)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.724641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.725904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.790258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ffc6f4d45494f6294514a9079188f8f', 'request_id': 'req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.793493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5900be0-062c-43ec-90ee-1b75757e9631 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.799197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ffc6f4d45494f6294514a9079188f8f', 'request_id': 'req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.806137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ffc6f4d45494f6294514a9079188f8f', 'request_id': 'req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.818878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] The token cache is being invalidate because project 1ffc6f4d45494f6294514a9079188f8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:04.819167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.819849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:04.821771 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1134/2273] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:04 2022] DELETE /identity/v3/projects/1ffc6f4d45494f6294514a9079188f8f => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:04.825717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.827063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.865880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.866908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.867344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.867774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.868573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.871595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.884904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:04.900719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1140/2274] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:04 2022] POST /identity/v3/users => generated 299 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:04.905034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.907549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:04.908759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8d8f5f-b231-4c4a-8844-1115ca2c2c95 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.920528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.921224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.921509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.921788 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] PATH_INFO: `/v3/users/1e9b25a2ed6447e5936b025ff390fe42` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.924589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e9b25a2ed6447e5936b025ff390fe42)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.926789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.928255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53759f4c-b86d-4d7c-8dcc-c8f8aca9e426 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.935570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.936197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.936480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.936755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.937577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:04.939974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:04.956796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1141/2275] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:04.961344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.974050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de529d91-ad66-435a-9080-aefd46d0c3ed admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:04.985861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:04.986534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:04.986821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:04.987088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] PATH_INFO: `/v3/projects/02999c86ddf24b2ca0334a693c9a8950/users/a1b8b19e76134765b04363f596c382dd/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:04.989955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:04.995389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e9b25a2ed6447e5936b025ff390fe42'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.003664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e9b25a2ed6447e5936b025ff390fe42'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.004413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e9b25a2ed6447e5936b025ff390fe42'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.005092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e9b25a2ed6447e5936b025ff390fe42'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.013930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02999c86ddf24b2ca0334a693c9a8950, user_id=a1b8b19e76134765b04363f596c382dd, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.016348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e9b25a2ed6447e5936b025ff390fe42'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.021459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.037386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1142/2276] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:04 2022] PUT /identity/v3/projects/02999c86ddf24b2ca0334a693c9a8950/users/a1b8b19e76134765b04363f596c382dd/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:05.041074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.042738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.044999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1135/2277] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:04 2022] DELETE /identity/v3/users/1e9b25a2ed6447e5936b025ff390fe42 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:05.123723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478988f7-490c-44d8-8d9f-ccb484812f93 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.153337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.155295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.225142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.226415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.227914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.228800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.229639 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.231627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.243778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efdf33-d629-4a1e-9532-b2437111edeb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.248311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1143/2278] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:05 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:05.249751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.253180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.253837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.254118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.254394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] PATH_INFO: `/v3/users/393f6e96e0c74640a9485147e305786f` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.255652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=393f6e96e0c74640a9485147e305786f)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.257954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.266016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de33b52-29db-4675-af64-c704641e4832 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.273548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaeece1-e1b7-4d30-8c73-9b4e2de94adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.274393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeece1-e1b7-4d30-8c73-9b4e2de94adc admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.274662 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeece1-e1b7-4d30-8c73-9b4e2de94adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.274921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeece1-e1b7-4d30-8c73-9b4e2de94adc admin admin] PATH_INFO: `/v3/projects/02999c86ddf24b2ca0334a693c9a8950/users/a1b8b19e76134765b04363f596c382dd/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.277379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaeece1-e1b7-4d30-8c73-9b4e2de94adc admin admin] RBAC: Authorizing `identity:create_grant(project_id=02999c86ddf24b2ca0334a693c9a8950, user_id=a1b8b19e76134765b04363f596c382dd, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.282008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaeece1-e1b7-4d30-8c73-9b4e2de94adc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.296894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1144/2279] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:05 2022] PUT /identity/v3/projects/02999c86ddf24b2ca0334a693c9a8950/users/a1b8b19e76134765b04363f596c382dd/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:05.307901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883b64fd-199f-4a3e-a55d-b980265e518d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.308211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883b64fd-199f-4a3e-a55d-b980265e518d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.308486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883b64fd-199f-4a3e-a55d-b980265e518d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.340326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '393f6e96e0c74640a9485147e305786f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.361578 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '393f6e96e0c74640a9485147e305786f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.362850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '393f6e96e0c74640a9485147e305786f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.364235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '393f6e96e0c74640a9485147e305786f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.364397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-883b64fd-199f-4a3e-a55d-b980265e518d None None] MFA Rules not processed for user `a1b8b19e76134765b04363f596c382dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:05.365781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-883b64fd-199f-4a3e-a55d-b980265e518d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.373222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '393f6e96e0c74640a9485147e305786f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.379665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:05.402911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1136/2280] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:05 2022] DELETE /identity/v3/users/393f6e96e0c74640a9485147e305786f => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:05.403889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.405251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.460453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:05.471590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1145/2281] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:05.472142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.472648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.473453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.495083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:05.496629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.505202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1146/2282] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:05.506189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.507521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.508937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7466f7-4852-4fcb-accd-ab9d091a13b7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.513697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.514338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.514624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.514902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] PATH_INFO: `/v3/projects/046f78fca72548068f4bb604b30e3238` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.516147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=046f78fca72548068f4bb604b30e3238)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.518627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.571101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100838ce-deb9-43f7-949f-3561c78ece02 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.582766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '046f78fca72548068f4bb604b30e3238', 'request_id': 'req-7a354590-6123-4a84-80aa-9854938dc8f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.590130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '046f78fca72548068f4bb604b30e3238', 'request_id': 'req-7a354590-6123-4a84-80aa-9854938dc8f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.597055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '046f78fca72548068f4bb604b30e3238', 'request_id': 'req-7a354590-6123-4a84-80aa-9854938dc8f9'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.609532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] The token cache is being invalidate because project 046f78fca72548068f4bb604b30e3238 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:05.609822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.610514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.612026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1137/2283] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:05 2022] DELETE /identity/v3/projects/046f78fca72548068f4bb604b30e3238 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:05.612791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.614095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.632138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.632890 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.633293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.633526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.635496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.680814 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a354590-6123-4a84-80aa-9854938dc8f9 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.684625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.685270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.685546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.685818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.687552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.704402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.706473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f42fc4-afb5-42f9-a4bb-a3838061ed2d service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.716558 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1147/2284] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:05 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:05.717968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6604ad11-e7aa-41d9-8e09-2688def27335 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.718261 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6604ad11-e7aa-41d9-8e09-2688def27335 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.718539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6604ad11-e7aa-41d9-8e09-2688def27335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.727337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:05.743425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6604ad11-e7aa-41d9-8e09-2688def27335 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:05.745079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6604ad11-e7aa-41d9-8e09-2688def27335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.767547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.769225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.778775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1138/2285] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:05.779573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.780921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.794647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee276d66-e50c-443b-909a-6a388cf257e2 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.799681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.800301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.800575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.800839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] PATH_INFO: `/v3/users/1759cacb633948e2b705e8bd42aaa923` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.802001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1759cacb633948e2b705e8bd42aaa923)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.803928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.806035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:05.826177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1148/2286] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:05.836914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.837238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.837476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.860105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:05.861391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.868023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1759cacb633948e2b705e8bd42aaa923'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.869489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1149/2287] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:05.875043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.879617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1759cacb633948e2b705e8bd42aaa923'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.883199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1759cacb633948e2b705e8bd42aaa923'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.884263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1759cacb633948e2b705e8bd42aaa923'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.888132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41cfa13-26de-49cf-b495-41dd06ebb9ec None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.894506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.895117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.895418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b19ef30b-a92d-427b-b21c-117f93ee05c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1759cacb633948e2b705e8bd42aaa923'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.895533 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.895996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.896711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:05.897730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.899188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.911470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1150/2288] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:05.915695 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.917455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.918682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1139/2289] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:05 2022] DELETE /identity/v3/users/1759cacb633948e2b705e8bd42aaa923 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:05.924082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.924368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.924665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.938116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab84d6-69d4-43d8-826e-8810a9de40f7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.949682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:05.951440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.013204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.014221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.014741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.015271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.016123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.018985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.057631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1151/2290] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/projects => generated 377 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:06.058310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1140/2291] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:06.061416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.063882 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.065665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.116479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:06.144477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10c10b-dd16-4e82-b4d2-1b67cf5b1f0d None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.160489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12ce78-3341-4af3-8106-9fe2cdb603ec admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.169578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.170576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.170883 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.172273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.173502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.175603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.202239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1152/2292] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:06 2022] POST /identity/v3/users => generated 289 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:06.202239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.209761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.213319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.213960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.214226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.214503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.232488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0adbab1-7f0a-4efd-b69f-b863244647a5 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.233728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.236085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.238951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.239894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.240377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.240780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] PATH_INFO: `/v3/projects/1c9a601330a9479e89ee25506a57aeff` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.242574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c9a601330a9479e89ee25506a57aeff)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.246214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.250375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1141/2293] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:06.251594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.264006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096cf2f-707a-4e12-a5ea-5f532a7d3d40 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.286556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.288242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.288984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.289277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.289941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.292529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.305484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1142/2294] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:06.309642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.322370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282f926-4845-49e1-b6ac-2d9e9961bcda admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.345665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:INSERT=4,keystone:DELETE=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:06.347861 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c9a601330a9479e89ee25506a57aeff', 'request_id': 'req-fb805e38-3098-45fc-96d6-c6645991c8a0'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.361301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c9a601330a9479e89ee25506a57aeff', 'request_id': 'req-fb805e38-3098-45fc-96d6-c6645991c8a0'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.376321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c9a601330a9479e89ee25506a57aeff', 'request_id': 'req-fb805e38-3098-45fc-96d6-c6645991c8a0'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.390924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] The token cache is being invalidate because project 1c9a601330a9479e89ee25506a57aeff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:06.391222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.391934 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.393949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1153/2295] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:06 2022] DELETE /identity/v3/projects/1c9a601330a9479e89ee25506a57aeff => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:06.397952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.400230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.423659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.424773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.425085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.426674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.428575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.430970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.447305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1143/2296] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:06.448565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.466437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.488444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e483c1b9-38b8-4ee3-a981-a3a9c617241d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.493424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e892e41-2504-444c-9076-bfc71ebd6a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.494153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e892e41-2504-444c-9076-bfc71ebd6a56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.494526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e892e41-2504-444c-9076-bfc71ebd6a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.494944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e892e41-2504-444c-9076-bfc71ebd6a56 admin admin] PATH_INFO: `/v3/projects/5d4d234fa2a44c58bb2f5f57b632969d/users/777f7f22edde4a808e3c3125e47e6956/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.512622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb805e38-3098-45fc-96d6-c6645991c8a0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.520383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e892e41-2504-444c-9076-bfc71ebd6a56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d4d234fa2a44c58bb2f5f57b632969d, user_id=777f7f22edde4a808e3c3125e47e6956, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.523909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e892e41-2504-444c-9076-bfc71ebd6a56 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.531136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.531824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.532079 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.532315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] PATH_INFO: `/v3/users/27684af6625944da98274dc3277c9e38` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.534745 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27684af6625944da98274dc3277c9e38)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.538063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.554326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1144/2297] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:06 2022] PUT /identity/v3/projects/5d4d234fa2a44c58bb2f5f57b632969d/users/777f7f22edde4a808e3c3125e47e6956/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:06.557427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.557722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.558217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.600852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] MFA Rules not processed for user `777f7f22edde4a808e3c3125e47e6956`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:06.602933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.636673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27684af6625944da98274dc3277c9e38'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.663545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27684af6625944da98274dc3277c9e38'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.674962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27684af6625944da98274dc3277c9e38'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.675589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:06.675878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27684af6625944da98274dc3277c9e38'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.676525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27684af6625944da98274dc3277c9e38'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.700195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1154/2298] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:06 2022] DELETE /identity/v3/users/27684af6625944da98274dc3277c9e38 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:06.701588 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.704800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1145/2299] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:06 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:06.717309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:06.720002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.720926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.814884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c050a-d5a3-480c-885c-f3374392f0a2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.838530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:06.846243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d944729a-9a7d-4898-9cc3-f842da3f4117 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.859135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.859957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.860255 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.860563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] PATH_INFO: `/v3/projects/0b10a783b984421788f1a32e3270694e` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.861769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b10a783b984421788f1a32e3270694e)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.863787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.885838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.886759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.887222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.887856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.958736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b10a783b984421788f1a32e3270694e', 'request_id': 'req-d58e03c1-77aa-4aa7-a593-939ec133f77d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.971706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.973407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.974543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b10a783b984421788f1a32e3270694e', 'request_id': 'req-d58e03c1-77aa-4aa7-a593-939ec133f77d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.985247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b10a783b984421788f1a32e3270694e', 'request_id': 'req-d58e03c1-77aa-4aa7-a593-939ec133f77d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:06.991502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1146/2300] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:07.019411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] The token cache is being invalidate because project 0b10a783b984421788f1a32e3270694e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:07.019411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.019411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.019411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1155/2301] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:06 2022] DELETE /identity/v3/projects/0b10a783b984421788f1a32e3270694e => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:07.026332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.042692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6426c5fb-1c5d-467b-8128-dc467c5e4de8 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.093919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.095590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.113242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:07.121588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.125113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.125640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.126721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] PATH_INFO: `/v3/users/a1b8b19e76134765b04363f596c382dd` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.129085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1b8b19e76134765b04363f596c382dd)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.131792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.179555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58e03c1-77aa-4aa7-a593-939ec133f77d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.187459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.189300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.190724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.191244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] PATH_INFO: `/v3/projects/b24151f6abf849a38789a40cc923e561` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.192277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1b8b19e76134765b04363f596c382dd'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.192857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b24151f6abf849a38789a40cc923e561)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.196823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.201521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1b8b19e76134765b04363f596c382dd'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.202377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1b8b19e76134765b04363f596c382dd'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.203264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1b8b19e76134765b04363f596c382dd'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.213962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-4bbee66e-4d5c-424a-b92c-c56d00a88482 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1b8b19e76134765b04363f596c382dd'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.237918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1147/2302] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:07 2022] DELETE /identity/v3/users/a1b8b19e76134765b04363f596c382dd => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:07.266390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b24151f6abf849a38789a40cc923e561', 'request_id': 'req-a0851714-e309-4a6d-a8d3-2e9727e0fc11'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.276351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b24151f6abf849a38789a40cc923e561', 'request_id': 'req-a0851714-e309-4a6d-a8d3-2e9727e0fc11'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.285121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b24151f6abf849a38789a40cc923e561', 'request_id': 'req-a0851714-e309-4a6d-a8d3-2e9727e0fc11'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.297253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] The token cache is being invalidate because project b24151f6abf849a38789a40cc923e561 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:07.298970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.299687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a0851714-e309-4a6d-a8d3-2e9727e0fc11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.301678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1156/2303] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:07 2022] DELETE /identity/v3/projects/b24151f6abf849a38789a40cc923e561 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:07.307134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.307409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.307683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.337769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:07.339244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.436156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1148/2304] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:07.447110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.447643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.447930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.470976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:07.472127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.480489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1157/2305] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:07.484570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.494820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad0bb96-28f7-40c3-9bb5-292bd808c36c None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.500514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.501224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.502020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.502295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.502948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:07.503253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.504855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.516500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1149/2306] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:07.521000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.530679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917e52c9-2987-47f6-8d55-a1023a57f9ba None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.536018 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.536683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.537110 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.537411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.538069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.540224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.545062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.546188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:07.547760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.563936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc10deb-fcce-4f7f-988d-6f58bdd023cc admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.570368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.571187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.571561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.571957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] PATH_INFO: `/v3/projects/02999c86ddf24b2ca0334a693c9a8950` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.573443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] RBAC: Authorizing `identity:delete_project(project_id=02999c86ddf24b2ca0334a693c9a8950)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.575524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.592633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1158/2307] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/projects => generated 381 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:07.597247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.648672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02999c86ddf24b2ca0334a693c9a8950', 'request_id': 'req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.657015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02999c86ddf24b2ca0334a693c9a8950', 'request_id': 'req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.659612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:07.665429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02999c86ddf24b2ca0334a693c9a8950', 'request_id': 'req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.669084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065db5a3-766a-457c-a433-de40e6d69c86 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.678761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] The token cache is being invalidate because project 02999c86ddf24b2ca0334a693c9a8950 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:07.679051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.679712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d20496f5-9dd0-4b52-aea5-31fa1a4a167e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:07.681565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1150/2308] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:07 2022] DELETE /identity/v3/projects/02999c86ddf24b2ca0334a693c9a8950 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:07.687252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41217cc-5b81-48c2-950d-ee7be73c22d5 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.687628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41217cc-5b81-48c2-950d-ee7be73c22d5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.687960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41217cc-5b81-48c2-950d-ee7be73c22d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.710349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d41217cc-5b81-48c2-950d-ee7be73c22d5 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:07.711693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d41217cc-5b81-48c2-950d-ee7be73c22d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.739806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.740344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.740628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.740958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.741703 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.743797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.765867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1159/2309] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/users => generated 291 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:07.770632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.772314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.781376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1151/2310] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:07.784647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb4f690-cd76-4aa9-b248-a9329a783913 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.789584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.789727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.789871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.790142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.790260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.790547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.790817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.791460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.793123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.804688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1160/2311] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:07.808772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.812092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:07.813335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.819475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ce0b4-eca5-496b-b5b5-15d973d0c09a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.822209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1152/2312] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:07.824140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.824738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.825061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.825429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.826070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.826379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.827677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.836637 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e835a2ac-0b28-451d-b4b4-aa9d18df196f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.843606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1161/2313] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:07.843777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.843777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.843777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.843777 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.844069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:07.844069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.844589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.844951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.856142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5711ef25-182a-4c5c-b202-dd2e3069472b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.856553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1153/2314] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:07.860840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.862545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805fb358-e8ba-4961-8929-b518cf38d2fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.863262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805fb358-e8ba-4961-8929-b518cf38d2fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.863545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805fb358-e8ba-4961-8929-b518cf38d2fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.863826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805fb358-e8ba-4961-8929-b518cf38d2fc admin admin] PATH_INFO: `/v3/projects/e087576431f744a0b23dd42ab0b8eb75/users/f981a0a86c8d42a6aed7b83726766ada/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.874852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873dcdee-bfea-4e7d-9562-daf0e6a4eb16 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.880060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.880662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.880971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.881224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805fb358-e8ba-4961-8929-b518cf38d2fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e087576431f744a0b23dd42ab0b8eb75, user_id=f981a0a86c8d42a6aed7b83726766ada, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.881333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.881857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.883733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.883916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805fb358-e8ba-4961-8929-b518cf38d2fc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:07.898161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1162/2315] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:07 2022] PUT /identity/v3/projects/e087576431f744a0b23dd42ab0b8eb75/users/f981a0a86c8d42a6aed7b83726766ada/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:07.908984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.909290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.909548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.914150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1154/2316] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/projects => generated 407 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:07.918282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:07.945031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] MFA Rules not processed for user `f981a0a86c8d42a6aed7b83726766ada`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:07.946602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:07.972274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:07.990394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f350052c-bf61-4ea3-b17f-9a40fcd728e2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:07.995507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:07.996116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:07.996399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:07.996675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:07.997338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:07.999316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.014041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1163/2317] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:08.018710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.026822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1155/2318] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:07 2022] POST /identity/v3/users => generated 304 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:08.028205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.030869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5900cb6e-79c2-471e-b3d1-bb5a04f7b03d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.036355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.036988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.037273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.037545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.038173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.040067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e19a0ef-ce1c-4fcd-b4a8-d5102a0a99c3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.040187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.050884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.051912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.052270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.052550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.053268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.054958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.066724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1156/2319] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:08.070994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.082098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6806b52d-14fc-4239-a027-112bcb6fce58 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.082241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1164/2320] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/projects => generated 381 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:08.086234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.110484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:08.154620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.155362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.155710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.156054 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.156762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.161759 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.163826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e349bf5a-d2ef-43a6-8608-a06f87bbb569 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.168956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.169577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.169858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.170131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.170765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.172826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.176718 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1157/2321] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:08.177750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.188521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da65cc44-16f0-46f2-89fc-53d94209edd3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.194797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.195450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.195751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.196025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] PATH_INFO: `/v3/projects/134252500ee5475dbd9c388fa7f0fcbc/users/434f7bc76acb46dcaef346b9afe3b783/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.198328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1165/2322] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/users => generated 297 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:08.202918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.216552 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c6c79-151f-4aae-a8dd-0946708bf304 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.221266 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=134252500ee5475dbd9c388fa7f0fcbc, user_id=434f7bc76acb46dcaef346b9afe3b783, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.224116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.224759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced0313c-b753-45b0-be31-af74fc750444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.225489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced0313c-b753-45b0-be31-af74fc750444 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.225839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced0313c-b753-45b0-be31-af74fc750444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.226187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced0313c-b753-45b0-be31-af74fc750444 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.226839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced0313c-b753-45b0-be31-af74fc750444 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.228504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced0313c-b753-45b0-be31-af74fc750444 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.239271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1158/2323] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:08 2022] PUT /identity/v3/projects/134252500ee5475dbd9c388fa7f0fcbc/users/434f7bc76acb46dcaef346b9afe3b783/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:08.244810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1166/2324] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:08.246334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.250659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.251014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.251333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.294427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] MFA Rules not processed for user `434f7bc76acb46dcaef346b9afe3b783`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:08.296233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:08.341714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5595cb44-39ca-43b9-858a-868839726c44 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.350299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.350983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.351397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.351680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] PATH_INFO: `/v3/projects/32cab68aa87e4ad38d4a33aa23017679/users/6a884811f404401eb00e019f9e742ac3/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.369921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] RBAC: Authorizing `identity:create_grant(project_id=32cab68aa87e4ad38d4a33aa23017679, user_id=6a884811f404401eb00e019f9e742ac3, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.373305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.391421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1159/2325] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:08 2022] PUT /identity/v3/projects/32cab68aa87e4ad38d4a33aa23017679/users/6a884811f404401eb00e019f9e742ac3/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:08.396598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.398784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1167/2326] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:08.400006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.421657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5af4561-0094-4b90-9ebc-e13c4f34b013 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.437729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:08.538607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a9e59-7f4c-4736-bc8e-7f501dd2368d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.542958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.543587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.544055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.544397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.545191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.546042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.546660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.546936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.547352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.547470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.548197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.550076 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.561173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1160/2327] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:08.565994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.580408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2036ea12-bad7-423b-8436-19955868916e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.589540 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5316e3df-4f80-4ef1-b6ad-8b86f97ad801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.590369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5316e3df-4f80-4ef1-b6ad-8b86f97ad801 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.590660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5316e3df-4f80-4ef1-b6ad-8b86f97ad801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.591023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5316e3df-4f80-4ef1-b6ad-8b86f97ad801 admin admin] PATH_INFO: `/v3/projects/32cab68aa87e4ad38d4a33aa23017679/users/6a884811f404401eb00e019f9e742ac3/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.593222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5316e3df-4f80-4ef1-b6ad-8b86f97ad801 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32cab68aa87e4ad38d4a33aa23017679, user_id=6a884811f404401eb00e019f9e742ac3, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.595469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1168/2328] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/projects => generated 407 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:08.596735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5316e3df-4f80-4ef1-b6ad-8b86f97ad801 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.599633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.619867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1161/2329] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:08 2022] PUT /identity/v3/projects/32cab68aa87e4ad38d4a33aa23017679/users/6a884811f404401eb00e019f9e742ac3/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:08.621507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.621801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.622085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.631471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:08.655476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] MFA Rules not processed for user `6a884811f404401eb00e019f9e742ac3`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:08.657215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:08.725918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa65234-2304-488c-840a-f14e6a261d7e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.780554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1162/2330] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:08.792612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.794976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:08.831030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.831694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.832607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.833026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.833752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.838386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.864153 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1169/2331] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/users => generated 310 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:08.865267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.872689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d071e89-17c2-4d89-8404-1ec784ef58fb None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.876507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb2e94c-ba80-4ae4-8fce-0cafd25e4e53 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.877960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.879040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.879040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.879233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.885567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.886186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.886475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.886753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.887412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.889418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.889539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.890932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.900899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1170/2332] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:08.902113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1163/2333] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:08.904812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.915705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20abeec1-5faa-4e13-8523-365f2e6b0aa0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:08.920801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:08.921516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.921798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.922077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] PATH_INFO: `/v3/projects/d8f15fcebbc34104afdb7b4316f5e78a/users/71ba06475f1b455b97016c649a32f2dd/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.939359 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8f15fcebbc34104afdb7b4316f5e78a, user_id=71ba06475f1b455b97016c649a32f2dd, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:08.942432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:08.952201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:08.960277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1171/2334] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:08 2022] PUT /identity/v3/projects/d8f15fcebbc34104afdb7b4316f5e78a/users/71ba06475f1b455b97016c649a32f2dd/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:08.964082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:08.976438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f552396-5724-4609-9f34-a64d2b8876bd service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.045431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62fa49e-5ec6-4b76-bfb5-90ffe3c1892f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.046207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fa49e-5ec6-4b76-bfb5-90ffe3c1892f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.046489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fa49e-5ec6-4b76-bfb5-90ffe3c1892f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.046761 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fa49e-5ec6-4b76-bfb5-90ffe3c1892f admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.047462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62fa49e-5ec6-4b76-bfb5-90ffe3c1892f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.049746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62fa49e-5ec6-4b76-bfb5-90ffe3c1892f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.061695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1164/2335] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:08 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:09.066764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:09.078762 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba139d-8672-47ef-ad23-b4c119ab3576 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.083515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.084164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.084441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.084716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] PATH_INFO: `/v3/projects/d8f15fcebbc34104afdb7b4316f5e78a/users/71ba06475f1b455b97016c649a32f2dd/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.086840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8f15fcebbc34104afdb7b4316f5e78a, user_id=71ba06475f1b455b97016c649a32f2dd, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.089526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.100980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1172/2336] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:09 2022] PUT /identity/v3/projects/d8f15fcebbc34104afdb7b4316f5e78a/users/71ba06475f1b455b97016c649a32f2dd/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:09.113153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.113447 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.113724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.143047 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] MFA Rules not processed for user `71ba06475f1b455b97016c649a32f2dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:09.144272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:09.158678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:09.222528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1165/2337] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:09 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:09.233593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:09.235893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:09.303587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4681468-5b1b-4d37-8909-3e46d7f3516c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.307594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.308200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.308479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.309224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.319270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.320911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.322144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:09.332192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1173/2338] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:09 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:09.386886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdedfde-c42a-4072-9070-779394b152f0 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.391857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.392491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.392770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.393075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] PATH_INFO: `/v3/users/f981a0a86c8d42a6aed7b83726766ada` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.394553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f981a0a86c8d42a6aed7b83726766ada)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.398008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.448119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f981a0a86c8d42a6aed7b83726766ada'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.455071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f981a0a86c8d42a6aed7b83726766ada'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.455764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f981a0a86c8d42a6aed7b83726766ada'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.456392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f981a0a86c8d42a6aed7b83726766ada'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.464267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e9cb186-67f9-4be9-aa42-507ffd810e3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f981a0a86c8d42a6aed7b83726766ada'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.487511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1166/2339] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:09 2022] DELETE /identity/v3/users/f981a0a86c8d42a6aed7b83726766ada => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:09.490507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:09.491556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.491868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.492168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.499348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13822e7d-d66c-4532-97b4-0a71088e995f service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.512234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:09.513599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:09.541196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:09.551345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.552107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.553470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.553470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.554666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:09.586747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1167/2340] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:09.596414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:09.598369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:09.645401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.646957 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.663661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1174/2341] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:09 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:09.668352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83538f3d-7be8-45ed-a7b9-3b690765ff04 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.672568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.673732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.674026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.674277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.685736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.687899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.702779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1168/2342] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:10.027977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.029794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.057392 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75a6139-ba23-4123-aa10-20c92eb9a45d service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.078441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.079122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.079491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.079812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] PATH_INFO: `/v3/projects/e087576431f744a0b23dd42ab0b8eb75` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.081536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e087576431f744a0b23dd42ab0b8eb75)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.083086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.133437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:10.142596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e087576431f744a0b23dd42ab0b8eb75', 'request_id': 'req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.151399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e087576431f744a0b23dd42ab0b8eb75', 'request_id': 'req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.157121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e087576431f744a0b23dd42ab0b8eb75', 'request_id': 'req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.168288 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] The token cache is being invalidate because project e087576431f744a0b23dd42ab0b8eb75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:10.168573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.169270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.170853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1175/2343] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:10 2022] DELETE /identity/v3/projects/e087576431f744a0b23dd42ab0b8eb75 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:10.174883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.184384 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da991e1-b36d-4966-b407-193579d7dac8 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.241052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.241752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.242030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.242351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] PATH_INFO: `/v3/users/6a884811f404401eb00e019f9e742ac3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.243457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a884811f404401eb00e019f9e742ac3)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.245592 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.292723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a884811f404401eb00e019f9e742ac3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.300269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a884811f404401eb00e019f9e742ac3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.300912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a884811f404401eb00e019f9e742ac3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.301526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a884811f404401eb00e019f9e742ac3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.308955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d4cded96-2c76-40cd-9a82-76946601f840 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a884811f404401eb00e019f9e742ac3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.319820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:10.327998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1169/2344] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:10 2022] DELETE /identity/v3/users/6a884811f404401eb00e019f9e742ac3 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:10.605267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.606734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.619988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e8c269-a5e5-4ecb-a1bc-0e7d1c868a2e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.675854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.676483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.676765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.677064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] PATH_INFO: `/v3/projects/32cab68aa87e4ad38d4a33aa23017679` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.678200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] RBAC: Authorizing `identity:delete_project(project_id=32cab68aa87e4ad38d4a33aa23017679)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.680118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.725173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32cab68aa87e4ad38d4a33aa23017679', 'request_id': 'req-da41cb23-dc13-4beb-bccf-273e9ca614af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.732240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32cab68aa87e4ad38d4a33aa23017679', 'request_id': 'req-da41cb23-dc13-4beb-bccf-273e9ca614af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.738170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32cab68aa87e4ad38d4a33aa23017679', 'request_id': 'req-da41cb23-dc13-4beb-bccf-273e9ca614af'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.748758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] The token cache is being invalidate because project 32cab68aa87e4ad38d4a33aa23017679 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:10.749077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.749733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-da41cb23-dc13-4beb-bccf-273e9ca614af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.751205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1176/2345] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:10 2022] DELETE /identity/v3/projects/32cab68aa87e4ad38d4a33aa23017679 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:10.756105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.756412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.756670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.777177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:10.778376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.839829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1170/2346] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:10.847560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.847845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.848121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.867636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:10.868838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.877775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1177/2347] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:10.881950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.891377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be754ce-5af9-4a98-9abd-19913316bf1a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.897016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.897615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.898006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.898284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.898931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:10.899232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.900815 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.913067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1171/2348] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:10.917433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.929952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f2ccde-7e67-4430-9658-004908a57fc2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.934680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.935280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.935561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.935839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.936465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.938425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.967067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1178/2349] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:10 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:10.972374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.982420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6050f168-72b9-4f5b-a10d-3cbd7290de7c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.048046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.048650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.048962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.049238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.049862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.051752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.073052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1172/2350] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:10 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:11.076960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.088067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f5ae62-5c08-47d0-9cb8-053332d6bb6d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.092985 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.093529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.094012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.094400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.104143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.105836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.119823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1179/2351] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:11 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:11.121336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.132259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf13dd3-1c3e-4cf5-8901-6999e731381c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.137172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab99eda-cdd0-45ac-8a21-c6ee02775539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.137798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab99eda-cdd0-45ac-8a21-c6ee02775539 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.138059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab99eda-cdd0-45ac-8a21-c6ee02775539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.138337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab99eda-cdd0-45ac-8a21-c6ee02775539 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.138975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab99eda-cdd0-45ac-8a21-c6ee02775539 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.140562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab99eda-cdd0-45ac-8a21-c6ee02775539 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.152137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1173/2352] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:11.156688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.168539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ba40a7-4859-4097-9967-24d8eaad49be admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.177062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.177650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.177929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.178201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] PATH_INFO: `/v3/projects/f879657e952f45449a3e68354ab7db07/users/526e5ed0c12d44338d929caf2e7ae45f/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.193468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f879657e952f45449a3e68354ab7db07, user_id=526e5ed0c12d44338d929caf2e7ae45f, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.196086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.198251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:11.209617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1180/2353] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:11 2022] PUT /identity/v3/projects/f879657e952f45449a3e68354ab7db07/users/526e5ed0c12d44338d929caf2e7ae45f/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:11.221055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51aa36-3ab1-4f88-98e3-a0748ec4609d None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.221341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51aa36-3ab1-4f88-98e3-a0748ec4609d None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.221614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da51aa36-3ab1-4f88-98e3-a0748ec4609d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.249181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-da51aa36-3ab1-4f88-98e3-a0748ec4609d None None] MFA Rules not processed for user `526e5ed0c12d44338d929caf2e7ae45f`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:11.250382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da51aa36-3ab1-4f88-98e3-a0748ec4609d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.297092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:11.317239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1174/2354] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:11 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:11.328709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.330485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.406196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5747f82-5e4b-4ddf-95c1-5ff3aedd9fe6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.409908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.410506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.410782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.411048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.419429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.420933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.430224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1181/2355] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:21.314001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:21.426463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:INSERT=1,keystone:SELECT=37 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:22.705312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12b9c2-7353-487f-9038-a15cae6eee34 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:22.705896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12b9c2-7353-487f-9038-a15cae6eee34 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:22.705896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12b9c2-7353-487f-9038-a15cae6eee34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:22.725407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3d12b9c2-7353-487f-9038-a15cae6eee34 None None] MFA Rules not processed for user `b38384d843d04503b69e5cb61af6c6ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:22.726615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3d12b9c2-7353-487f-9038-a15cae6eee34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:22.789301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1175/2356] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:23.612729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:23.614581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:23.698642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d078a7-2700-4777-baf9-d6e9bac39088 service glance] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:23.707830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:23.707830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:23.707830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:23.707830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] PATH_INFO: `/v3/users/434f7bc76acb46dcaef346b9afe3b783` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:23.709746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] RBAC: Authorizing `identity:delete_user(user_id=434f7bc76acb46dcaef346b9afe3b783)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:23.711832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:23.779833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '434f7bc76acb46dcaef346b9afe3b783'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:23.796730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '434f7bc76acb46dcaef346b9afe3b783'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:23.815555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '434f7bc76acb46dcaef346b9afe3b783'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:23.816385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '434f7bc76acb46dcaef346b9afe3b783'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:23.817520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '434f7bc76acb46dcaef346b9afe3b783'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:23.836910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1182/2357] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:23 2022] DELETE /identity/v3/users/434f7bc76acb46dcaef346b9afe3b783 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:23.842117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:23.843157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:23.843527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:23.864692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:23.866298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:23.941123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1176/2358] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:23.951547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:23.953413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:24.022757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8855d81a-2e34-4ac3-97e7-b5261f62a06f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:24.027074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:24.027684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:24.028020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:24.028262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:24.039229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:24.040923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:24.052344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1183/2359] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:24.333675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:24.335367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:24.348044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf9e5b5-a241-421a-8a07-6b89d0ca7ba7 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:24.353010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:24.353640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:24.353917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:24.354187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] PATH_INFO: `/v3/projects/134252500ee5475dbd9c388fa7f0fcbc` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:24.355522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=134252500ee5475dbd9c388fa7f0fcbc)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:24.358288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:24.407782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '134252500ee5475dbd9c388fa7f0fcbc', 'request_id': 'req-603c3743-e818-4593-8a22-a6d4d24c30f7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.414730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '134252500ee5475dbd9c388fa7f0fcbc', 'request_id': 'req-603c3743-e818-4593-8a22-a6d4d24c30f7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.421102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '134252500ee5475dbd9c388fa7f0fcbc', 'request_id': 'req-603c3743-e818-4593-8a22-a6d4d24c30f7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.432612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] The token cache is being invalidate because project 134252500ee5475dbd9c388fa7f0fcbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:24.433207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.433843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.435315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1177/2360] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:24 2022] DELETE /identity/v3/projects/134252500ee5475dbd9c388fa7f0fcbc => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:24.440135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:24.449659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65259ae-3f24-4123-881e-75ab34091994 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:24.467396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:24.515776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:24.518333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:24.518611 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:24.518882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] PATH_INFO: `/v3/users/71ba06475f1b455b97016c649a32f2dd` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:24.520175 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71ba06475f1b455b97016c649a32f2dd)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:24.522120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:24.568210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71ba06475f1b455b97016c649a32f2dd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.580650 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71ba06475f1b455b97016c649a32f2dd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.588427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71ba06475f1b455b97016c649a32f2dd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.589230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71ba06475f1b455b97016c649a32f2dd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.589888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0b41a41e-74d1-4cfa-99ed-669ddd044817 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71ba06475f1b455b97016c649a32f2dd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:24.603922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1184/2361] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:24 2022] DELETE /identity/v3/users/71ba06475f1b455b97016c649a32f2dd => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:24.887005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:24.889710 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:24.903555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603c3743-e818-4593-8a22-a6d4d24c30f7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:24.932072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:24.968972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:24.970188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:24.970467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:24.970742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] PATH_INFO: `/v3/projects/d8f15fcebbc34104afdb7b4316f5e78a` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:24.971864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8f15fcebbc34104afdb7b4316f5e78a)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:24.973807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.025833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8f15fcebbc34104afdb7b4316f5e78a', 'request_id': 'req-bbcb3ebe-c60c-4212-ba58-2853b0113f73'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:25.034323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8f15fcebbc34104afdb7b4316f5e78a', 'request_id': 'req-bbcb3ebe-c60c-4212-ba58-2853b0113f73'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:25.040722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8f15fcebbc34104afdb7b4316f5e78a', 'request_id': 'req-bbcb3ebe-c60c-4212-ba58-2853b0113f73'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:25.051921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] The token cache is being invalidate because project d8f15fcebbc34104afdb7b4316f5e78a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:25.052213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:25.052861 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-bbcb3ebe-c60c-4212-ba58-2853b0113f73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:25.054371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1178/2362] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:24 2022] DELETE /identity/v3/projects/d8f15fcebbc34104afdb7b4316f5e78a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:25.059451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.059926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.060249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.082573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:25.083857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:25.157114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1185/2363] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:25.158961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.159249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.159523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.178909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:25.180104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:25.187919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1179/2364] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:25.192769 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.202481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff306a10-5733-4877-a20f-9a030650c894 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.207595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.208180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.208453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.208721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.209394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:25.209694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.211268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.223123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1186/2365] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:25.227195 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.237553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd244135-19f9-44bd-b0b8-5adf23b69dcc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.242519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.243112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.243394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.243669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.244310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.246252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.275334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1180/2366] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:25.279596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.290220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08104126-39d8-49cd-b62b-5b9e448872e2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.315975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:25.353066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.353599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.353880 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.354154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.354794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.356706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.379779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1187/2367] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/users => generated 288 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:25.384986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.398430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffc1b6-53fc-420b-a281-40dede9cd33e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.405102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.405705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.405978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.406248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.406885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.408507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.418480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1181/2368] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:25.423070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.432887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1af4cc-4144-4d25-8a6b-ae8820377ef2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.437538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0541959-3010-4120-b503-8b1a2c70c62c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.438680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0541959-3010-4120-b503-8b1a2c70c62c admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.438963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0541959-3010-4120-b503-8b1a2c70c62c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.439230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0541959-3010-4120-b503-8b1a2c70c62c admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.440117 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0541959-3010-4120-b503-8b1a2c70c62c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.441713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0541959-3010-4120-b503-8b1a2c70c62c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.451478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1188/2369] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:25.455593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.465563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7f227-e0ea-40ed-bc94-ede754bd0014 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.470239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.470875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.471156 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.471427 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] PATH_INFO: `/v3/projects/2f94839bac5748b9ba9a963064f48e01/users/6f491a7376234c11974bff0973c10828/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.486394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f94839bac5748b9ba9a963064f48e01, user_id=6f491a7376234c11974bff0973c10828, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.489126 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.502875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1182/2370] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:25 2022] PUT /identity/v3/projects/2f94839bac5748b9ba9a963064f48e01/users/6f491a7376234c11974bff0973c10828/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:25.519546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.519831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.520108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.548998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] MFA Rules not processed for user `6f491a7376234c11974bff0973c10828`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:25.550211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:25.618473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1189/2371] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:25.622807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.684673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:25.691114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b9fe6b-0ff4-4472-9386-42099f0bbf9a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.696478 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.697138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.697421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.697695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.698328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.700214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.726341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1183/2372] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/projects => generated 375 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:25.730452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.741070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c96e8-1d49-47e3-bb01-495a0208e27e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.799815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.800427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.800707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.801014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.801808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.803736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.824513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1190/2373] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:25 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:25.828527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.839692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0951b-be97-49e1-a3cf-1ff96327a009 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.844751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.845463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.845749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.846019 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.846660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.848231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.858693 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1184/2374] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:25.862778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.872775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb75d78f-8eea-471f-ada6-be04e4974cc1 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.877818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:25.878456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:25.878733 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:25.879004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] PATH_INFO: `/v3/projects/c0b862d322034d3eaabfe72aff5bdfc9/users/5cbeae55b34b4f36af060b3e1603b4ee/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:25.893529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0b862d322034d3eaabfe72aff5bdfc9, user_id=5cbeae55b34b4f36af060b3e1603b4ee, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:25.896842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:25.926351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1191/2375] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:25 2022] PUT /identity/v3/projects/c0b862d322034d3eaabfe72aff5bdfc9/users/5cbeae55b34b4f36af060b3e1603b4ee/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:25.930353 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:25.941531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b620fd08-d69c-478d-9010-25b8eb3cbe5e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:25.999691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13274dc3-2d74-4059-a73d-58e0b26487d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:26.000291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13274dc3-2d74-4059-a73d-58e0b26487d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:26.000571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13274dc3-2d74-4059-a73d-58e0b26487d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:26.000858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13274dc3-2d74-4059-a73d-58e0b26487d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:26.001521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13274dc3-2d74-4059-a73d-58e0b26487d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:26.003091 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13274dc3-2d74-4059-a73d-58e0b26487d6 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:26.019372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1185/2376] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:26.024221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:26.029747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:26.036808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0950be6b-29a5-4387-97df-e253aff5a460 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:26.042263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:26.042958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:26.043268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:26.043540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] PATH_INFO: `/v3/projects/c0b862d322034d3eaabfe72aff5bdfc9/users/5cbeae55b34b4f36af060b3e1603b4ee/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:26.046375 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0b862d322034d3eaabfe72aff5bdfc9, user_id=5cbeae55b34b4f36af060b3e1603b4ee, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:26.050099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:26.062666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1192/2377] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:26 2022] PUT /identity/v3/projects/c0b862d322034d3eaabfe72aff5bdfc9/users/5cbeae55b34b4f36af060b3e1603b4ee/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:26.073297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:26.073583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:26.073865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:26.102434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] MFA Rules not processed for user `5cbeae55b34b4f36af060b3e1603b4ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:26.103670 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:26.173562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1186/2378] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:26 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:26.184941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:26.186697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:26.245436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54be8a6-7939-4bed-ae83-76268da91ab6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:26.249671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e630e64-b2f4-41f5-a494-7140dddb2400 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:26.250279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e630e64-b2f4-41f5-a494-7140dddb2400 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:26.250596 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e630e64-b2f4-41f5-a494-7140dddb2400 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:26.251144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e630e64-b2f4-41f5-a494-7140dddb2400 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:26.311220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e630e64-b2f4-41f5-a494-7140dddb2400 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:26.312663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e630e64-b2f4-41f5-a494-7140dddb2400 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:26.322237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1193/2379] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:26 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:29.021203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.096157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:29.111103 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2120fa6c-b7af-41c7-8281-d4883240404a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.116588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.117461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.117730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.118006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] PATH_INFO: `/v3/users/6f491a7376234c11974bff0973c10828` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.119141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f491a7376234c11974bff0973c10828)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.121134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:29.192080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f491a7376234c11974bff0973c10828'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.202530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f491a7376234c11974bff0973c10828'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.203269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f491a7376234c11974bff0973c10828'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.204425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f491a7376234c11974bff0973c10828'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.217110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f491a7376234c11974bff0973c10828'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.259514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1187/2380] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:29 2022] DELETE /identity/v3/users/6f491a7376234c11974bff0973c10828 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:29.264295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b127158-b89b-4247-898f-60737f429be0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.266616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b127158-b89b-4247-898f-60737f429be0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.267345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b127158-b89b-4247-898f-60737f429be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.295444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2b127158-b89b-4247-898f-60737f429be0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:29.296895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2b127158-b89b-4247-898f-60737f429be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.364684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1194/2381] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:29.375642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.377297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.440430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a22f2a1-ab40-43d1-ad64-cecf07935dd0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.445287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.445928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.446290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.446602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.457444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.458873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:29.468830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1188/2382] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:29.755741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b127158-b89b-4247-898f-60737f429be0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.757315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2b127158-b89b-4247-898f-60737f429be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.790313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b127158-b89b-4247-898f-60737f429be0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.795327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.795859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.796196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.796483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] PATH_INFO: `/v3/projects/2f94839bac5748b9ba9a963064f48e01` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.798259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f94839bac5748b9ba9a963064f48e01)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.801080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:29.859510 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:29.872730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f94839bac5748b9ba9a963064f48e01', 'request_id': 'req-829367d8-0583-402a-987d-c4eb01ea91c1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.883137 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f94839bac5748b9ba9a963064f48e01', 'request_id': 'req-829367d8-0583-402a-987d-c4eb01ea91c1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.891717 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f94839bac5748b9ba9a963064f48e01', 'request_id': 'req-829367d8-0583-402a-987d-c4eb01ea91c1'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.906506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] The token cache is being invalidate because project 2f94839bac5748b9ba9a963064f48e01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:29.907285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.908037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.909623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1195/2383] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:29 2022] DELETE /identity/v3/projects/2f94839bac5748b9ba9a963064f48e01 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:29.914190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.925122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f033cb-90ac-4b8f-b9b3-88b4cb23b368 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:30.064012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:30.066622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:30.067124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:30.067316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] PATH_INFO: `/v3/users/5cbeae55b34b4f36af060b3e1603b4ee` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:30.069619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5cbeae55b34b4f36af060b3e1603b4ee)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:30.074838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:30.106063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:30.148223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5cbeae55b34b4f36af060b3e1603b4ee'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.156431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5cbeae55b34b4f36af060b3e1603b4ee'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.165360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5cbeae55b34b4f36af060b3e1603b4ee'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.166507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5cbeae55b34b4f36af060b3e1603b4ee'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.185955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f140ab39-aa85-4113-a788-5d5c4dd9e482 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5cbeae55b34b4f36af060b3e1603b4ee'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.215839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1189/2384] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:29 2022] DELETE /identity/v3/users/5cbeae55b34b4f36af060b3e1603b4ee => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:30.623401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:30.625080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:30.647364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829367d8-0583-402a-987d-c4eb01ea91c1 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:30.724587 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:30.726449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:30.727021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:30.727568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] PATH_INFO: `/v3/projects/c0b862d322034d3eaabfe72aff5bdfc9` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:30.732397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0b862d322034d3eaabfe72aff5bdfc9)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:30.732397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:30.791495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0b862d322034d3eaabfe72aff5bdfc9', 'request_id': 'req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.800649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0b862d322034d3eaabfe72aff5bdfc9', 'request_id': 'req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.807448 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0b862d322034d3eaabfe72aff5bdfc9', 'request_id': 'req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.821338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] The token cache is being invalidate because project c0b862d322034d3eaabfe72aff5bdfc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:30.821631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.822340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e5f12d2a-cf80-48db-8862-fe64ca03e1f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:30.824131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1196/2385] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:30 2022] DELETE /identity/v3/projects/c0b862d322034d3eaabfe72aff5bdfc9 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:30.830326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:30.830619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:30.830908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:30.859269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:30.861071 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:30.922060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1190/2386] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:30.930286 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:30.930573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:30.930853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:30.952246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:30.965357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:30.968387 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1197/2387] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:30.968496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:30.979270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2d4cdf-31f2-4281-84ae-e8b38bbc75fe None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:30.984273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:30.985193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:30.985536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:30.986047 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:30.986779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:30.987170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:30.988852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.002104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1191/2388] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:31.007904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.018713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73efd6d6-32fd-4afc-a764-1c81ddaaa32a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.023503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.024274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.024555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.024826 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.025487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.027398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.056597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1198/2389] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:31.060750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.072727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4eb3d5f-baca-4f8e-8a8e-72fb28198294 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.158407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.159122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.159403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.159953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.160609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.162575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.183997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1192/2390] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/users => generated 285 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:31.188665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.199696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c98cf0-aa85-42d3-8162-90afe1b5f258 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.204586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.205213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.205489 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.205756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.206390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.207979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.218275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1199/2391] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:31.222575 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.232741 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f57da2-dfa2-4564-ac3d-961ea35f3a06 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.238781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4216ef10-d7dd-4fde-a8fa-af259ec6a234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.239673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4216ef10-d7dd-4fde-a8fa-af259ec6a234 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.240198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4216ef10-d7dd-4fde-a8fa-af259ec6a234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.240484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4216ef10-d7dd-4fde-a8fa-af259ec6a234 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.241174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4216ef10-d7dd-4fde-a8fa-af259ec6a234 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.243115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4216ef10-d7dd-4fde-a8fa-af259ec6a234 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.245629 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:31.261251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1193/2392] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:31 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:31.268239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.286915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b875c4-2667-4b86-9d27-2db0fcaf8f48 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.293007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.293901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.294172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.294564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] PATH_INFO: `/v3/projects/d9a4439df0bc4de7965947e066e41be4/users/b35814f63cb34cd29bea888b4f61133f/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.311487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9a4439df0bc4de7965947e066e41be4, user_id=b35814f63cb34cd29bea888b4f61133f, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.314207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.316612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:31.328688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1200/2393] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:31 2022] PUT /identity/v3/projects/d9a4439df0bc4de7965947e066e41be4/users/b35814f63cb34cd29bea888b4f61133f/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:31.338616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.338906 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.339180 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.367457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] MFA Rules not processed for user `b35814f63cb34cd29bea888b4f61133f`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:31.368682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.431375 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1194/2394] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:31.443688 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.445635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.504791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f71793-cb9f-4515-bcbd-92f7680b93d2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.508801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee31e26c-da5f-41f9-b4cb-b618ec33089c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.509547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee31e26c-da5f-41f9-b4cb-b618ec33089c service glance] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.509854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee31e26c-da5f-41f9-b4cb-b618ec33089c service glance] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.510123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee31e26c-da5f-41f9-b4cb-b618ec33089c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.519411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee31e26c-da5f-41f9-b4cb-b618ec33089c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.520857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee31e26c-da5f-41f9-b4cb-b618ec33089c service glance] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.530744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1201/2395] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:31 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:31.684122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.685778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.769237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ddd4fc-4f77-46bc-8249-14984517ebb5 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.774572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.775340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.775593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.775865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] PATH_INFO: `/v3/users/777f7f22edde4a808e3c3125e47e6956` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.777046 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=777f7f22edde4a808e3c3125e47e6956)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.780022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.840079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '777f7f22edde4a808e3c3125e47e6956'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.863036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '777f7f22edde4a808e3c3125e47e6956'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.863854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '777f7f22edde4a808e3c3125e47e6956'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.864627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '777f7f22edde4a808e3c3125e47e6956'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.873167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '777f7f22edde4a808e3c3125e47e6956'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.895152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1195/2396] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:31 2022] DELETE /identity/v3/users/777f7f22edde4a808e3c3125e47e6956 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:31.899444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.899962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.899962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.924687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:31.926109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.018080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1202/2397] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:32.028900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.034069 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.061782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:32.143832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3ccf9d-d694-4241-95e1-196a74000d4a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.149116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce22fe-4604-40b1-b9d6-7ed635a94279 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.149984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce22fe-4604-40b1-b9d6-7ed635a94279 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.150468 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce22fe-4604-40b1-b9d6-7ed635a94279 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.150793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce22fe-4604-40b1-b9d6-7ed635a94279 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.165672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce22fe-4604-40b1-b9d6-7ed635a94279 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.167308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce22fe-4604-40b1-b9d6-7ed635a94279 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.180264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1196/2398] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:32.604927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.606765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.621064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8bfbb3-4313-4d06-8a14-95edaf186e8d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.628084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.629051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.629333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.629657 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] PATH_INFO: `/v3/projects/5d4d234fa2a44c58bb2f5f57b632969d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.630812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d4d234fa2a44c58bb2f5f57b632969d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.632670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.708905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d4d234fa2a44c58bb2f5f57b632969d', 'request_id': 'req-256b30a8-472b-4276-a8e7-1951d726523e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.721801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d4d234fa2a44c58bb2f5f57b632969d', 'request_id': 'req-256b30a8-472b-4276-a8e7-1951d726523e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.732005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d4d234fa2a44c58bb2f5f57b632969d', 'request_id': 'req-256b30a8-472b-4276-a8e7-1951d726523e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.776842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] The token cache is being invalidate because project 5d4d234fa2a44c58bb2f5f57b632969d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:32.776842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.777241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-256b30a8-472b-4276-a8e7-1951d726523e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.780363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1203/2399] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:32 2022] DELETE /identity/v3/projects/5d4d234fa2a44c58bb2f5f57b632969d => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:32.784972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.785339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.785640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.823295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:32.824630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.894263 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1197/2400] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:32.903444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcced16-4472-4885-a76d-e6657397fefc None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.904348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcced16-4472-4885-a76d-e6657397fefc None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.904784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcced16-4472-4885-a76d-e6657397fefc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.925289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-adcced16-4472-4885-a76d-e6657397fefc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:32.926499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-adcced16-4472-4885-a76d-e6657397fefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.934727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1204/2401] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:32.939146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.948898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc747d2-17bb-4754-b6e9-4cfb19818d68 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.954584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.955182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.955464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.955735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.956374 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:32.956673 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.958289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.971174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1198/2402] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:32.975665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcced16-4472-4885-a76d-e6657397fefc None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.985525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcced16-4472-4885-a76d-e6657397fefc None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.990758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.991384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.991859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.992146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.993285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.994923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.008637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:33.027894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1205/2403] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:32 2022] POST /identity/v3/projects => generated 375 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:33.033304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.050619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c9ec3-8f2c-4521-a1b7-b8a94ff06e89 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.126621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.128109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.128391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.128663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.129705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.131641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.153728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1199/2404] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/users => generated 288 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:33.158092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.179445 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cc8202-ee08-4c8b-9c6a-193ff53abe22 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.184487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.185199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.185542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.185899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.186599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.188759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.199949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1206/2405] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:33.203995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.213773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b8a605-27b4-4a02-bb9b-1e868bf186a7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.218340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ea8b4-4c7e-4ca6-918e-33f5793bbd03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.218958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ea8b4-4c7e-4ca6-918e-33f5793bbd03 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.219238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ea8b4-4c7e-4ca6-918e-33f5793bbd03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.219506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ea8b4-4c7e-4ca6-918e-33f5793bbd03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.220137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014ea8b4-4c7e-4ca6-918e-33f5793bbd03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.221799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014ea8b4-4c7e-4ca6-918e-33f5793bbd03 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.232221 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1200/2406] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:33.237507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.250486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0510aad-cf46-4d93-a4fa-4c960a717627 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.256973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.257570 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.257846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.258115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] PATH_INFO: `/v3/projects/91cc1d88749b49e59b40e14c31f52997/users/87c3dea095c0415eb8ecc89e48889b23/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.274913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91cc1d88749b49e59b40e14c31f52997, user_id=87c3dea095c0415eb8ecc89e48889b23, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.278244 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.290989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1207/2407] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:33 2022] PUT /identity/v3/projects/91cc1d88749b49e59b40e14c31f52997/users/87c3dea095c0415eb8ecc89e48889b23/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:33.301134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.301449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.301721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.330812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] MFA Rules not processed for user `87c3dea095c0415eb8ecc89e48889b23`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:33.332017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.336170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:33.409811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1201/2408] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:33.420598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.422316 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.481978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129cfaf7-9ee6-4775-b574-4a4d78fb38c7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.485683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9c0df-0fdc-4317-8657-dadf18f24478 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.486292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9c0df-0fdc-4317-8657-dadf18f24478 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.486572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9c0df-0fdc-4317-8657-dadf18f24478 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.486842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9c0df-0fdc-4317-8657-dadf18f24478 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.495074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9c0df-0fdc-4317-8657-dadf18f24478 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.496507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9c0df-0fdc-4317-8657-dadf18f24478 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.505937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1208/2409] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:33.551524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.553072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.647820 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fda634-bddb-476b-b917-145cfc29552a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.652486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.653242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.653527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.653802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] PATH_INFO: `/v3/users/526e5ed0c12d44338d929caf2e7ae45f` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.654949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=526e5ed0c12d44338d929caf2e7ae45f)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.657570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.730573 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '526e5ed0c12d44338d929caf2e7ae45f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.744143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '526e5ed0c12d44338d929caf2e7ae45f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.745187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '526e5ed0c12d44338d929caf2e7ae45f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.752172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '526e5ed0c12d44338d929caf2e7ae45f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.766256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '526e5ed0c12d44338d929caf2e7ae45f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.787940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1202/2410] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:33 2022] DELETE /identity/v3/users/526e5ed0c12d44338d929caf2e7ae45f => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:33.792270 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.792560 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.792956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.813438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:33.814954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.895196 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1209/2411] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:33.906964 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.908752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.946342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:33.981606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db4ebf-0dcd-42e1-b67b-931e316e2bc3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.985854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93234f1a-ff5c-4eb0-9520-5c85ea29d6e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.986457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93234f1a-ff5c-4eb0-9520-5c85ea29d6e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.986737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93234f1a-ff5c-4eb0-9520-5c85ea29d6e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.987017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93234f1a-ff5c-4eb0-9520-5c85ea29d6e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.996021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93234f1a-ff5c-4eb0-9520-5c85ea29d6e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.997505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93234f1a-ff5c-4eb0-9520-5c85ea29d6e8 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.008763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1203/2412] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:34.465334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.467233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.480241 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dc9496-7547-4cc6-9810-9e309ee7e5e2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.485660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.487466 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.487746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.488011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] PATH_INFO: `/v3/projects/f879657e952f45449a3e68354ab7db07` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.489182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f879657e952f45449a3e68354ab7db07)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.491083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.537378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f879657e952f45449a3e68354ab7db07', 'request_id': 'req-89af9370-6ff9-4444-a271-9a13d91ba035'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.540831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:34.544607 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f879657e952f45449a3e68354ab7db07', 'request_id': 'req-89af9370-6ff9-4444-a271-9a13d91ba035'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.551829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f879657e952f45449a3e68354ab7db07', 'request_id': 'req-89af9370-6ff9-4444-a271-9a13d91ba035'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.562851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] The token cache is being invalidate because project f879657e952f45449a3e68354ab7db07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:34.563138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.563766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-89af9370-6ff9-4444-a271-9a13d91ba035 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.565518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1210/2413] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:34 2022] DELETE /identity/v3/projects/f879657e952f45449a3e68354ab7db07 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:34.570322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.570775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.570866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.591512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:34.592764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.653261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1204/2414] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:34.661351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.661664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.661942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.682199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:34.683420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.692271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1211/2415] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:34.696417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.705902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a349b456-366e-4178-b35a-c68bbb9c9ed6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.710552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.711090 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.711372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.711643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.712283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:34.712586 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.714204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.725398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1205/2416] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:34.729423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.739192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e77e157-fef0-45ba-bb98-2aa88b67afbd None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.743863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.744454 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.744755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.745057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.745719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.747629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.774553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1212/2417] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:34.778786 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.789118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faecac95-be90-4855-a42c-2e45da35e5da admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.854935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.855576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.855862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.856169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.856803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.858756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.882692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1206/2418] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/users => generated 285 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:34.883956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.894424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35ab13-0968-4aa4-82e7-4d09023cd681 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.899328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.899931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.900208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.900482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.901160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.903235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.921112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1213/2419] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:34.922377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.933005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54396177-e688-4ad4-923b-e0c1945e3195 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.938463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8741fc9e-89e8-4881-be33-c0258c9f98cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.939067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8741fc9e-89e8-4881-be33-c0258c9f98cf admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.939349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8741fc9e-89e8-4881-be33-c0258c9f98cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.939878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8741fc9e-89e8-4881-be33-c0258c9f98cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.940515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8741fc9e-89e8-4881-be33-c0258c9f98cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.942115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8741fc9e-89e8-4881-be33-c0258c9f98cf admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.959571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1207/2420] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:34.966357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.971092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.971676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.974811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.979385 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bebde8-72a5-4191-88a9-d92d35a5ef59 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.990425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a098ab3-d3ad-46cb-8f65-1694e5d8454d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.991124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a098ab3-d3ad-46cb-8f65-1694e5d8454d admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.991402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a098ab3-d3ad-46cb-8f65-1694e5d8454d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.991710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a098ab3-d3ad-46cb-8f65-1694e5d8454d admin admin] PATH_INFO: `/v3/projects/2260a975996746859072523b7fc6964f/users/b68e096c644f40869e8429e0fe378b75/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.996685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] MFA Rules not processed for user `87c3dea095c0415eb8ecc89e48889b23`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:34.997950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:35.007212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a098ab3-d3ad-46cb-8f65-1694e5d8454d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2260a975996746859072523b7fc6964f, user_id=b68e096c644f40869e8429e0fe378b75, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:35.009900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a098ab3-d3ad-46cb-8f65-1694e5d8454d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:35.025238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1214/2421] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:34 2022] PUT /identity/v3/projects/2260a975996746859072523b7fc6964f/users/b68e096c644f40869e8429e0fe378b75/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:35.027481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:35.036534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac612e33-4912-42d6-b5a5-15c9d437027c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:35.036816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac612e33-4912-42d6-b5a5-15c9d437027c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:35.037123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac612e33-4912-42d6-b5a5-15c9d437027c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:35.055385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1208/2422] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:35.066029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ac612e33-4912-42d6-b5a5-15c9d437027c None None] MFA Rules not processed for user `b68e096c644f40869e8429e0fe378b75`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:35.067231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ac612e33-4912-42d6-b5a5-15c9d437027c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:35.133346 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1215/2423] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:35 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:35.144935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:35.146397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:35.217449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63abef-6a0d-4741-9b44-d88d5ef82c1c None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:35.273858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:35.274481 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:35.274743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:35.275175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:35.286895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:35.288916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:35.297513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1209/2424] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:35 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:39.973805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8658c70-df97-454c-940a-36a667794ff1 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:39.974427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8658c70-df97-454c-940a-36a667794ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:39.974427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8658c70-df97-454c-940a-36a667794ff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:40.005889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e8658c70-df97-454c-940a-36a667794ff1 None None] MFA Rules not processed for user `53c020af94d64bac81df7837a86586b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:40.007107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e8658c70-df97-454c-940a-36a667794ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:40.081803 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1216/2425] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:39 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:45.293400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=56 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:46.208530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:46.209978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:46.269625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f652a5-5440-4927-b0a3-54e1fcb2bc94 service glance] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:46.276648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:46.276648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:46.276648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:46.276648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:46.277655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:46.338611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:46.340050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf78fc9c-a758-487b-a083-04e2fe44736d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:46.350131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1210/2426] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:46 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:50.069342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=90,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:53.144705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:53.146057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:53.146709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:53.176102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] MFA Rules not processed for user `f16c1cde815c4a2580cd5ebf7b3ed76d`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:53.177666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:53.276815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1217/2427] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:13:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:56.345796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=50 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:03.269341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=25 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:05.851268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed9f4ae-9dbe-43e2-bedc-350a4c1ff8a6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.851996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed9f4ae-9dbe-43e2-bedc-350a4c1ff8a6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.852288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed9f4ae-9dbe-43e2-bedc-350a4c1ff8a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.875611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0ed9f4ae-9dbe-43e2-bedc-350a4c1ff8a6 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:05.876946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0ed9f4ae-9dbe-43e2-bedc-350a4c1ff8a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:05.937389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1211/2428] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:14:05 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:05.984942 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.986365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:06.057227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004d80a7-e22b-4828-904a-20299e097a45 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:06.062082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:06.062687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:06.062963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:06.063245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:06.071673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:06.073348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:06.082856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1218/2429] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:14:06.654411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc8bb3e-baff-4fd2-88a4-0073e7681828 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:06.654926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc8bb3e-baff-4fd2-88a4-0073e7681828 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:06.658405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc8bb3e-baff-4fd2-88a4-0073e7681828 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:06.684129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-1fc8bb3e-baff-4fd2-88a4-0073e7681828 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:06.686284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1fc8bb3e-baff-4fd2-88a4-0073e7681828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:06.703513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1212/2430] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 29 02:14:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:06.717014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:06.728524 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635244d-022a-4228-82d5-d884add37b0a service swift] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:06.733555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:06.734138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:06.734417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:06.734689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:06.742449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:06.744013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:06.751869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1219/2431] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:14:07.012214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8054ac19-feca-4815-ac7f-88e8f3091bac None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:07.012676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8054ac19-feca-4815-ac7f-88e8f3091bac None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:07.012769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8054ac19-feca-4815-ac7f-88e8f3091bac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:07.032626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8054ac19-feca-4815-ac7f-88e8f3091bac None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:07.033862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8054ac19-feca-4815-ac7f-88e8f3091bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:07.040890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1213/2432] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:14:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:07.052456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:07.063417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d755b153-a9a4-4136-9fc0-d4541d2c652d service swift] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:07.068718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:07.069344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:07.069626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:07.069903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:07.078327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:07.079906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:07.087829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1220/2433] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:14:07.435863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:07.436214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:07.436520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:07.455978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:07.457191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:07.464257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1214/2434] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 29 02:14:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:07.476360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:07.487380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38863c37-0748-4c16-a9cc-8ea7b2cffb3d service swift] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:07.493405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff8c9f6-4ebf-4953-bf15-cb30e05affb6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:07.494053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff8c9f6-4ebf-4953-bf15-cb30e05affb6 service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:07.494540 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff8c9f6-4ebf-4953-bf15-cb30e05affb6 service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:07.495045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff8c9f6-4ebf-4953-bf15-cb30e05affb6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:07.504469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff8c9f6-4ebf-4953-bf15-cb30e05affb6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:07.506100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff8c9f6-4ebf-4953-bf15-cb30e05affb6 service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:07.514223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1221/2435] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:14:12.859362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:12.860838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:12.939857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c08d92-eea9-40e6-b0ee-edf6c81a9f58 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:12.944473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:12.945284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:12.945564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:12.945831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] PATH_INFO: `/v3/users/87c3dea095c0415eb8ecc89e48889b23` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:12.946967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=87c3dea095c0415eb8ecc89e48889b23)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:12.948918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.006015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87c3dea095c0415eb8ecc89e48889b23'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.019141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87c3dea095c0415eb8ecc89e48889b23'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.020185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87c3dea095c0415eb8ecc89e48889b23'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.020676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87c3dea095c0415eb8ecc89e48889b23'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.036511 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87c3dea095c0415eb8ecc89e48889b23'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.061323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1215/2436] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:12 2022] DELETE /identity/v3/users/87c3dea095c0415eb8ecc89e48889b23 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:13.065189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.066054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.066713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.116796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:13.118077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:13.220839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1222/2437] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:13.229416 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.231077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:13.239816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:13.310668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d97555-d3a8-471d-abf9-a96b0ace69b0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.315136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d572e77-911a-4add-8151-8863a10611c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.315818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d572e77-911a-4add-8151-8863a10611c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.316169 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d572e77-911a-4add-8151-8863a10611c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.316510 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d572e77-911a-4add-8151-8863a10611c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.327088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d572e77-911a-4add-8151-8863a10611c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.328633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d572e77-911a-4add-8151-8863a10611c5 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.338513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1216/2438] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:13.509300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.511338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:13.524436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad80963-37d5-44e2-a7cd-1dde59eb9b8e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.529208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.530027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.530237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.531113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] PATH_INFO: `/v3/projects/91cc1d88749b49e59b40e14c31f52997` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.532306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=91cc1d88749b49e59b40e14c31f52997)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.534267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.586767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:13.588136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91cc1d88749b49e59b40e14c31f52997', 'request_id': 'req-cf11461d-1964-4c26-8666-aad6001e0e0b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.596249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91cc1d88749b49e59b40e14c31f52997', 'request_id': 'req-cf11461d-1964-4c26-8666-aad6001e0e0b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.601907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91cc1d88749b49e59b40e14c31f52997', 'request_id': 'req-cf11461d-1964-4c26-8666-aad6001e0e0b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.613353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] The token cache is being invalidate because project 91cc1d88749b49e59b40e14c31f52997 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:13.613644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.614281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cf11461d-1964-4c26-8666-aad6001e0e0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.615763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1223/2439] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:13 2022] DELETE /identity/v3/projects/91cc1d88749b49e59b40e14c31f52997 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:13.621085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.621378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.621652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.641437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:13.642684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:13.700422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1217/2440] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:13.708917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.709295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.709579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.730353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:13.731564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:13.739415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1224/2441] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:13.743504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.753116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a177eb-6f7c-4b65-a227-51ef85687266 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.757633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.758224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.758503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.758773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.759409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:13.759713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.761348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.773293 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1218/2442] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:13.777409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.786996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636b4f3a-27be-43a0-82c1-676c997f0770 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.791561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.792143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.792420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.792685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.793344 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.795301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.824930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1225/2443] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:13 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:13.829668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.839677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a97441-2e1e-4779-8e66-ebbf628b1807 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.895146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.895850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.896014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.896287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.896971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.898892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.920493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1219/2444] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:13 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:13.924685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.934873 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9356b-3548-4573-8b7a-829d2a0c9181 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.940230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.940813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.941118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.941389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.942053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.943636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.954067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1226/2445] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:13.958108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.967948 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6caf09e-ae55-4842-9084-49bd684f5b91 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.972556 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be7ee43-d8dd-444a-83ab-f2dc95f6681a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.973183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7ee43-d8dd-444a-83ab-f2dc95f6681a admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.973465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7ee43-d8dd-444a-83ab-f2dc95f6681a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.973738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be7ee43-d8dd-444a-83ab-f2dc95f6681a admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.974406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be7ee43-d8dd-444a-83ab-f2dc95f6681a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.975981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be7ee43-d8dd-444a-83ab-f2dc95f6681a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.986913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1220/2446] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:13.990959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:14.001835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7449ba8-6aad-4307-9e8e-fe72038f1f51 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:14.006394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:14.007027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:14.007304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:14.007575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] PATH_INFO: `/v3/projects/18eeabc669304148930130efa1737cbb/users/ef9f29b86ed947919bcb5d6166ac0641/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:14.022210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18eeabc669304148930130efa1737cbb, user_id=ef9f29b86ed947919bcb5d6166ac0641, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:14.024840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:14.037546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1227/2447] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:13 2022] PUT /identity/v3/projects/18eeabc669304148930130efa1737cbb/users/ef9f29b86ed947919bcb5d6166ac0641/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:14.042579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:14.042862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:14.043133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:14.062656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:14.063879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:14.073388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:14.127407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1221/2448] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:14.137012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:14.138874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:14.205694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071cc29e-d275-42ac-948c-aa4206de3427 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:14.209065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15781ac5-1a5c-4069-9686-9ceefe4e7c12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:14.209664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15781ac5-1a5c-4069-9686-9ceefe4e7c12 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:14.209940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15781ac5-1a5c-4069-9686-9ceefe4e7c12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:14.210206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15781ac5-1a5c-4069-9686-9ceefe4e7c12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:14.219922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15781ac5-1a5c-4069-9686-9ceefe4e7c12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:14.221395 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15781ac5-1a5c-4069-9686-9ceefe4e7c12 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:14.230988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1228/2449] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:17.674588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:17.676182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:17.693458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b62bd02-6201-434b-8bc6-5dbed6b6e778 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:17.699864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:17.700677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:17.701123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:17.701762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] PATH_INFO: `/v3/users/b68e096c644f40869e8429e0fe378b75` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:17.703922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b68e096c644f40869e8429e0fe378b75)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:17.706403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:17.778782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b68e096c644f40869e8429e0fe378b75'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:17.787721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b68e096c644f40869e8429e0fe378b75'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:17.788688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b68e096c644f40869e8429e0fe378b75'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:17.789420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b68e096c644f40869e8429e0fe378b75'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:17.797358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b68e096c644f40869e8429e0fe378b75'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:17.817008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1222/2450] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:17 2022] DELETE /identity/v3/users/b68e096c644f40869e8429e0fe378b75 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:17.821488 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:17.822036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:17.822135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:17.845239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:17.846961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:17.922312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1229/2451] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:17.932006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:17.933656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:17.948500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:17.950126 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:17.965813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd3cb0-1ab5-40f8-8d74-10bddce57a53 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:17.971764 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:17.972407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:17.973034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:17.973034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] PATH_INFO: `/v3/users/b35814f63cb34cd29bea888b4f61133f` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:17.974293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b35814f63cb34cd29bea888b4f61133f)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:17.976229 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:17.983014 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:18.027312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4f5ee8-d514-4661-ad29-e35f18b6b31a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.031830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796bc0a3-5544-4174-95ae-8e9677f0ac4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.032524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796bc0a3-5544-4174-95ae-8e9677f0ac4c service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.032941 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796bc0a3-5544-4174-95ae-8e9677f0ac4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.033284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796bc0a3-5544-4174-95ae-8e9677f0ac4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.039164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b35814f63cb34cd29bea888b4f61133f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.043066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796bc0a3-5544-4174-95ae-8e9677f0ac4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.044618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796bc0a3-5544-4174-95ae-8e9677f0ac4c service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:18.053615 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b35814f63cb34cd29bea888b4f61133f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.056292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1223/2452] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:18.069004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b35814f63cb34cd29bea888b4f61133f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.069787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b35814f63cb34cd29bea888b4f61133f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.070982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b35814f63cb34cd29bea888b4f61133f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.092054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1230/2453] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:17 2022] DELETE /identity/v3/users/b35814f63cb34cd29bea888b4f61133f => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:18.093913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.094155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.094471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.132037 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:18.135606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.193890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:18.279310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1224/2454] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:18.292665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:18.292665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.388549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8063-d56f-43d7-8a52-013211694c7d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.394130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b81e6ab-f614-42b5-afe4-e758e2e92e28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.394754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b81e6ab-f614-42b5-afe4-e758e2e92e28 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.395039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b81e6ab-f614-42b5-afe4-e758e2e92e28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.395414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b81e6ab-f614-42b5-afe4-e758e2e92e28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.409400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b81e6ab-f614-42b5-afe4-e758e2e92e28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.411287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b81e6ab-f614-42b5-afe4-e758e2e92e28 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:18.424131 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1231/2455] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:18.505582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:18.507480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.532679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb20d3e-8a86-43e2-bfe8-403016d9daad None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.538036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.538678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.538963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.539237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] PATH_INFO: `/v3/projects/2260a975996746859072523b7fc6964f` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.552725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2260a975996746859072523b7fc6964f)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.558871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:18.641903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2260a975996746859072523b7fc6964f', 'request_id': 'req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.651868 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2260a975996746859072523b7fc6964f', 'request_id': 'req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.668916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2260a975996746859072523b7fc6964f', 'request_id': 'req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.693187 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] The token cache is being invalidate because project 2260a975996746859072523b7fc6964f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:18.693855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.700149 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-194a4c3f-7429-4b7c-a3d4-831c7e38e44c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.708288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1225/2456] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:18 2022] DELETE /identity/v3/projects/2260a975996746859072523b7fc6964f => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:18.710208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.710494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.710933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.755232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:18.757373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.902875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1232/2457] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:18.905445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.905774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.906023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.930229 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:18.931775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.942646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1226/2458] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:18.947711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:18.965200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383413f2-6f97-4e57-b3c0-bd7ec4757f01 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.977874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.978516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.978801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.979080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.979746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:18.980054 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.984599 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.000927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1233/2459] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:19.003504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.013724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4445a73e-814c-45a3-b353-149e7f781f74 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.019930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.020737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.021758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.021758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.022256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.025225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.069382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1227/2460] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/projects => generated 399 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:19.075104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.104463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332f50d-7ac5-41e2-a0fc-564ffa6de36c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.126814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.207213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.208003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.208369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.208734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.209537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.211800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.246668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1234/2461] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/users => generated 300 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:19.250063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.277981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391daf13-881e-4b1d-b1b4-d266fdc67344 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.298745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.299388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.300827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.301259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.306367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.318271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.336451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.338315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.353702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e04d1b1-0a80-41bb-a2c1-73c3a18c7c26 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.366284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1228/2462] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:19.372232 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.380238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.392442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.393006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.393460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] PATH_INFO: `/v3/projects/d9a4439df0bc4de7965947e066e41be4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.395702 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9a4439df0bc4de7965947e066e41be4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.398319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.404203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8013c7e-99f4-4553-8250-1796e7ba3164 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.422173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.422725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.423003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.423407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.424056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.430212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.476817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1229/2463] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:19.482098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.508807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0e948-f70e-4f24-b588-acac76c9a7d8 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.532563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9a4439df0bc4de7965947e066e41be4', 'request_id': 'req-566fd958-2865-4bbd-8666-e68fd49f9131'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.551280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9a4439df0bc4de7965947e066e41be4', 'request_id': 'req-566fd958-2865-4bbd-8666-e68fd49f9131'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.565652 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9a4439df0bc4de7965947e066e41be4', 'request_id': 'req-566fd958-2865-4bbd-8666-e68fd49f9131'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.601351 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] The token cache is being invalidate because project d9a4439df0bc4de7965947e066e41be4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:19.601674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.603425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-566fd958-2865-4bbd-8666-e68fd49f9131 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.610684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1235/2464] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:19 2022] DELETE /identity/v3/projects/d9a4439df0bc4de7965947e066e41be4 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:19.612412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e9d404-306e-48f5-9099-13589761bf0c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.612709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e9d404-306e-48f5-9099-13589761bf0c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.613040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e9d404-306e-48f5-9099-13589761bf0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.671358 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d7e9d404-306e-48f5-9099-13589761bf0c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:19.673593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d7e9d404-306e-48f5-9099-13589761bf0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.693218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e7a2cd-d0e9-446e-9909-6e0aa0759f79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.693912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e7a2cd-d0e9-446e-9909-6e0aa0759f79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.694193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e7a2cd-d0e9-446e-9909-6e0aa0759f79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.695766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e7a2cd-d0e9-446e-9909-6e0aa0759f79 admin admin] PATH_INFO: `/v3/projects/56f4ce8b9cee4754be380500431fb6b4/users/5138f0dc69d7470596874a35bc61130c/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.704113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.734981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e7a2cd-d0e9-446e-9909-6e0aa0759f79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56f4ce8b9cee4754be380500431fb6b4, user_id=5138f0dc69d7470596874a35bc61130c, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.737562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e7a2cd-d0e9-446e-9909-6e0aa0759f79 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.760599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1230/2465] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:19 2022] PUT /identity/v3/projects/56f4ce8b9cee4754be380500431fb6b4/users/5138f0dc69d7470596874a35bc61130c/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:19.771292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.771581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.771966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.795526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1236/2466] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:19.798032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.798325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.798606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.810714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] MFA Rules not processed for user `5138f0dc69d7470596874a35bc61130c`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:19.812077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.841257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:19.843029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.899515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.946522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1231/2467] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:19.956639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.958803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.995372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1237/2468] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:19.999875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.022941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc3ab2c-e37f-4dda-9da4-01d42a4f613b None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.033293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.034151 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.034514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.035481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.036293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:20.036678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.050565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.080981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1238/2469] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:14:20.082182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.102134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0d3e4c-47d3-48e6-80b5-7fcec9071d04 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.110529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca632be-7864-42b3-83d6-53c430ea4109 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.115697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.115984 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.116647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.117004 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.117120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.117415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.117917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.118192 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.118327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.120285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.130341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.132028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.145505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1232/2470] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:20.197473 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1239/2471] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/projects => generated 389 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:20.201655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.218253 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3df4449-84f1-45be-90b0-8ae2c2dd36ec service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.366804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.367449 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.367736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.370323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.371002 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.378993 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.437113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1233/2472] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/users => generated 295 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:20.441492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.455453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4ce58-c90f-4a74-851a-3813c2c26563 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.472158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.472816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.473123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.473401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.474056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.476072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.512092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1240/2473] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:20.513100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.527940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf7b177-3419-49c8-8379-ba35f9481465 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.554850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea3efd-ad6e-47ab-b6a2-c8a16b65a785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.555445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea3efd-ad6e-47ab-b6a2-c8a16b65a785 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.555732 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea3efd-ad6e-47ab-b6a2-c8a16b65a785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.556007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea3efd-ad6e-47ab-b6a2-c8a16b65a785 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.556669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea3efd-ad6e-47ab-b6a2-c8a16b65a785 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.574605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea3efd-ad6e-47ab-b6a2-c8a16b65a785 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.579933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:20.625056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1234/2474] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:20.630545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.648171 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63302ab8-0d18-4a0a-9796-58d24ee7b17c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.663913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.665763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.665763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.666644 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] PATH_INFO: `/v3/projects/a848d7b405cc4ec6bbaf07e573c783c8/users/c38fff7931774d42a6fb7134c759411f/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.698926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] RBAC: Authorizing `identity:create_grant(project_id=a848d7b405cc4ec6bbaf07e573c783c8, user_id=c38fff7931774d42a6fb7134c759411f, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.701682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.720891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1241/2475] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:20 2022] PUT /identity/v3/projects/a848d7b405cc4ec6bbaf07e573c783c8/users/c38fff7931774d42a6fb7134c759411f/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:20.733862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d803160f-2148-4b5a-9038-5211be80c432 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.734372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d803160f-2148-4b5a-9038-5211be80c432 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.734458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d803160f-2148-4b5a-9038-5211be80c432 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.770260 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d803160f-2148-4b5a-9038-5211be80c432 None None] MFA Rules not processed for user `c38fff7931774d42a6fb7134c759411f`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:20.771975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d803160f-2148-4b5a-9038-5211be80c432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.913094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1235/2476] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:20.934797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.936767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:21.051608 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e7b6a-7d71-4129-bb3b-4c1f98ed4ced admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:21.058927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:21.059692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.060047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.060613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.088606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:21.091452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:21.112928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1242/2477] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:21.236038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.236660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.237191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.281880 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] MFA Rules not processed for user `ef9f29b86ed947919bcb5d6166ac0641`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:21.283202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:21.448659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1236/2478] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:21.452433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:21.457825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:21.630636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2e6e91-cfcf-4038-b4eb-1ba2ec78a7f7 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:21.635786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:21.637100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.637678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.638026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.638839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:21.641202 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:21.647794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:21.681680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1243/2479] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:21 2022] POST /identity/v3/projects => generated 371 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:21.689212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:21.703115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb4108-9118-49df-8db4-63e28ca2f82a None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:21.782658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:21.783699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.783979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.784247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.784901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:21.787114 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:21.817407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1237/2480] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:21 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:21.821767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:21.840697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad65a727-975a-4844-b482-85c26a2cbbe6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:21.847572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:21.848348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.848634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.849235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.849606 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:21.851221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:21.862689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1244/2481] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:21 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:21.866867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:21.873272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:21.877623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babf4fe-2c01-408f-a2f9-38026179ac95 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:21.883245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:21.883968 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.884256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.884533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] PATH_INFO: `/v3/projects/1af0fde6012a448dbb0a35db223f8aaa/users/854f55ef9cc44cf687a17f1de857ff6b/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.901864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1af0fde6012a448dbb0a35db223f8aaa, user_id=854f55ef9cc44cf687a17f1de857ff6b, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:21.904719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:21.919866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1238/2482] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:21 2022] PUT /identity/v3/projects/1af0fde6012a448dbb0a35db223f8aaa/users/854f55ef9cc44cf687a17f1de857ff6b/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:21.924342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:21.934973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0254f4-c477-4604-b857-641f43696d1b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:21.991836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc224479-76d7-46cc-ac66-9e6414224c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:21.992386 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc224479-76d7-46cc-ac66-9e6414224c43 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:21.992663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc224479-76d7-46cc-ac66-9e6414224c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:21.992971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc224479-76d7-46cc-ac66-9e6414224c43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:21.993600 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc224479-76d7-46cc-ac66-9e6414224c43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:21.995180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc224479-76d7-46cc-ac66-9e6414224c43 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:22.008281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1245/2483] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:21 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:22.012337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:22.023778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f08aa23-be6e-4c06-acc7-640eda7270a5 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:22.028539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:22.029234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:22.029518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:22.029795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] PATH_INFO: `/v3/projects/1af0fde6012a448dbb0a35db223f8aaa/users/854f55ef9cc44cf687a17f1de857ff6b/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:22.032564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=1af0fde6012a448dbb0a35db223f8aaa, user_id=854f55ef9cc44cf687a17f1de857ff6b, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:22.035225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:22.051917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1239/2484] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:22 2022] PUT /identity/v3/projects/1af0fde6012a448dbb0a35db223f8aaa/users/854f55ef9cc44cf687a17f1de857ff6b/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:26.622283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b18a3cc-1c75-453a-bdad-907d234da1e3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:26.623287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b18a3cc-1c75-453a-bdad-907d234da1e3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:26.624287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b18a3cc-1c75-453a-bdad-907d234da1e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:26.685428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5b18a3cc-1c75-453a-bdad-907d234da1e3 None None] MFA Rules not processed for user `854f55ef9cc44cf687a17f1de857ff6b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:26.687127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5b18a3cc-1c75-453a-bdad-907d234da1e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:26.809964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1246/2485] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:26 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:26.824482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:26.826560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:26.940143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33611ec-9849-42c5-9eb0-f20a25c9e1bd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:26.947192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e098f-c460-47d8-ba3b-33bd29de4aec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:26.947823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e098f-c460-47d8-ba3b-33bd29de4aec service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:26.948105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e098f-c460-47d8-ba3b-33bd29de4aec service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:26.948431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e098f-c460-47d8-ba3b-33bd29de4aec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:27.011604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e098f-c460-47d8-ba3b-33bd29de4aec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:27.013583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e098f-c460-47d8-ba3b-33bd29de4aec service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:27.023574 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1240/2486] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:28.130971 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dbfeed-c54c-4bae-9643-226200401861 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.131541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dbfeed-c54c-4bae-9643-226200401861 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.131541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dbfeed-c54c-4bae-9643-226200401861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.151206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-50dbfeed-c54c-4bae-9643-226200401861 None None] MFA Rules not processed for user `ef9f29b86ed947919bcb5d6166ac0641`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:28.152500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-50dbfeed-c54c-4bae-9643-226200401861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:28.160916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1247/2487] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:29.510436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f66d6b-a69e-4ff9-b211-9b6fe53e23bc None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:29.511382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f66d6b-a69e-4ff9-b211-9b6fe53e23bc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:29.511787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f66d6b-a69e-4ff9-b211-9b6fe53e23bc None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:29.513705 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1241/2488] 10.222.0.27 () {54 vars in 961 bytes} [Thu Sep 29 02:14:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:14:29.519102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:29.519673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:29.519673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:29.542085 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:29.544084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:29.606766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:29.630915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1248/2489] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Sep 29 02:14:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:29.648342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f66d6b-a69e-4ff9-b211-9b6fe53e23bc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:29.671358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f66d6b-a69e-4ff9-b211-9b6fe53e23bc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:29.677803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:29.678699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:29.678982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:29.679254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:29.681612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:29.697466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:29.699264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb721194-4886-45bd-86c7-d126c0515d24 service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:29.710298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1242/2490] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:30.037039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:30.038453 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:30.109020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed4378b-e0f5-4912-b012-79811d8f04fe None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:30.116915 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:30.117878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:30.117878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:30.118072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] PATH_INFO: `/v3/users/5138f0dc69d7470596874a35bc61130c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:30.119572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=5138f0dc69d7470596874a35bc61130c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:30.122100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:30.178680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5138f0dc69d7470596874a35bc61130c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.191121 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5138f0dc69d7470596874a35bc61130c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.197918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5138f0dc69d7470596874a35bc61130c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.198624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5138f0dc69d7470596874a35bc61130c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.199228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5138f0dc69d7470596874a35bc61130c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.213234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1249/2491] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:30 2022] DELETE /identity/v3/users/5138f0dc69d7470596874a35bc61130c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:30.218334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:30.218627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:30.218902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:30.238918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:30.240159 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:30.289649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:30.304257 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1243/2492] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:30.314925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:30.316348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:30.372860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5616eae9-1fea-43bc-b3c0-3bf8cc6c87fc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:30.376991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:30.377590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:30.377895 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:30.378164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:30.387376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:30.388810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:30.399218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1250/2493] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:30.682545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:30.685335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:30.702219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175909f6-8b21-4690-b088-208b2e9b3de0 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:30.713995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:30.714972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:30.715376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:30.715806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] PATH_INFO: `/v3/projects/56f4ce8b9cee4754be380500431fb6b4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:30.717284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56f4ce8b9cee4754be380500431fb6b4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:30.719407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:30.792329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56f4ce8b9cee4754be380500431fb6b4', 'request_id': 'req-85c88b59-5415-492c-91f9-656ceb645fb3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.804558 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56f4ce8b9cee4754be380500431fb6b4', 'request_id': 'req-85c88b59-5415-492c-91f9-656ceb645fb3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.812991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56f4ce8b9cee4754be380500431fb6b4', 'request_id': 'req-85c88b59-5415-492c-91f9-656ceb645fb3'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.828524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] The token cache is being invalidate because project 56f4ce8b9cee4754be380500431fb6b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:30.829225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.829961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-85c88b59-5415-492c-91f9-656ceb645fb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:30.831588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1244/2494] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:30 2022] DELETE /identity/v3/projects/56f4ce8b9cee4754be380500431fb6b4 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:40.394939 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=85,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:40.825291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=31,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:42.964840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:42.966277 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:42.978907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960cd534-cc0a-4668-9e6f-bc04703a9556 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.034180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.034805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.035088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.035364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] PATH_INFO: `/v3/users/c38fff7931774d42a6fb7134c759411f` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.036474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c38fff7931774d42a6fb7134c759411f)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.038456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.086661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c38fff7931774d42a6fb7134c759411f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.099523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c38fff7931774d42a6fb7134c759411f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.106272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c38fff7931774d42a6fb7134c759411f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.106933 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c38fff7931774d42a6fb7134c759411f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.108062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c38fff7931774d42a6fb7134c759411f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.121257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1251/2495] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:42 2022] DELETE /identity/v3/users/c38fff7931774d42a6fb7134c759411f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:43.125812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.126274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.126372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.145083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:43.146291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:43.204857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1245/2496] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:43.214617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.216019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:43.274965 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5a2677-223e-4979-927e-34a45388a948 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.278642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd0ac8-4480-4a3f-9d37-5262b391e896 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.279767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd0ac8-4480-4a3f-9d37-5262b391e896 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.280047 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd0ac8-4480-4a3f-9d37-5262b391e896 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.280321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd0ac8-4480-4a3f-9d37-5262b391e896 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.289484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bd0ac8-4480-4a3f-9d37-5262b391e896 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.290925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bd0ac8-4480-4a3f-9d37-5262b391e896 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.300909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1252/2497] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:43.566354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.567804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:43.580611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d3764d-a18c-45f7-b273-54b0669868fa None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.585411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.586177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.586323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.586596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] PATH_INFO: `/v3/projects/a848d7b405cc4ec6bbaf07e573c783c8` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.587842 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a848d7b405cc4ec6bbaf07e573c783c8)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.589841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.638087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a848d7b405cc4ec6bbaf07e573c783c8', 'request_id': 'req-36936e73-abe6-4cae-a9ba-89f92e20ad26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.645859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a848d7b405cc4ec6bbaf07e573c783c8', 'request_id': 'req-36936e73-abe6-4cae-a9ba-89f92e20ad26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.652359 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a848d7b405cc4ec6bbaf07e573c783c8', 'request_id': 'req-36936e73-abe6-4cae-a9ba-89f92e20ad26'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.664928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] The token cache is being invalidate because project a848d7b405cc4ec6bbaf07e573c783c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:43.665245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.665927 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-36936e73-abe6-4cae-a9ba-89f92e20ad26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:43.668051 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1246/2498] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:43 2022] DELETE /identity/v3/projects/a848d7b405cc4ec6bbaf07e573c783c8 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:43.674238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.674526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.674804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.695542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:43.697078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:43.713919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:43.758103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1253/2499] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:43.766165 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.766455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.766731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.786845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:43.788043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:43.796454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1247/2500] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:43.800612 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.810218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe3df35-22da-46fd-902c-be8fc77e3c9c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.814969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.815555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.815832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.816101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.816734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:43.817062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.818638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.829648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1254/2501] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:43.833767 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.843450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a344484-9fa5-4499-8ff3-ab4d582f5e62 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.848189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.848813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.849128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.849399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.850125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.851915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.878916 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1248/2502] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:43.883104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.893692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be7cc3f-613b-456e-adda-49a5d269e33f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.948172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.948786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.949081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.949354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.949975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.951929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.973068 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1255/2503] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/users => generated 292 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:43.977065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.987892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da03b9f3-42e4-4d58-b2c2-a99288e3c765 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.992450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.993094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.993379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.993651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.994288 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.995863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.005576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1249/2504] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:44.009483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.019260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b1f75-c695-40d0-a7d4-3911d89ffc71 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.023808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4632018a-4175-4bf7-bd3a-c6c394fca8d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.024394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4632018a-4175-4bf7-bd3a-c6c394fca8d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.024674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4632018a-4175-4bf7-bd3a-c6c394fca8d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.024974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4632018a-4175-4bf7-bd3a-c6c394fca8d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.025610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4632018a-4175-4bf7-bd3a-c6c394fca8d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.027203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4632018a-4175-4bf7-bd3a-c6c394fca8d5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.038180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1256/2505] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:44 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:44.042244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.052618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fd296b-3d47-48fa-8d3a-3ec65c6e1429 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.057262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.057895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.058176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.058453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] PATH_INFO: `/v3/projects/40f01307928c46918e93db66b6473733/users/75c0e91cec2b455f82574ed9f110be92/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.073101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40f01307928c46918e93db66b6473733, user_id=75c0e91cec2b455f82574ed9f110be92, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.075747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.088183 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1250/2506] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:44 2022] PUT /identity/v3/projects/40f01307928c46918e93db66b6473733/users/75c0e91cec2b455f82574ed9f110be92/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:44.093621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf52b580-1d64-423c-a76b-1f863875537f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.093908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf52b580-1d64-423c-a76b-1f863875537f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.094182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf52b580-1d64-423c-a76b-1f863875537f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.113381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-cf52b580-1d64-423c-a76b-1f863875537f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:44.114575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-cf52b580-1d64-423c-a76b-1f863875537f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.172909 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1257/2507] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:44.182662 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.184393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.206724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:44.243910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8bc60-498d-4be6-b392-a1d8449db941 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.247550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.248207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.248486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.248756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.257639 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.259064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.268880 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1251/2508] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:49.020167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.020853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.020853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.091711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] MFA Rules not processed for user `75c0e91cec2b455f82574ed9f110be92`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:49.093258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:49.115791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:49.185796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1258/2509] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:49.191525 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:49.203725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca798f-a2ff-4b04-b21d-6c94cce2462d service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:49.210430 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:49.211049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.211939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.212168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.212813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:49.215537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:49.277529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1252/2510] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:49 2022] POST /identity/v3/projects => generated 383 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:49.278796 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:49.313132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307c266-7e5e-4259-969f-8ba4ce19795a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:49.430494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:49.431300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.431687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.432057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.432805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:49.434960 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:49.486817 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1259/2511] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:49 2022] POST /identity/v3/users => generated 298 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:49.490429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:49.503035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca2236-ebdd-4143-a57c-4d3566b77b13 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:49.510231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:49.511937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.512746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.514197 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.516740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:49.518691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:49.537668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1253/2512] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:49.538263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:49.551721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa36ed2-c9f0-4b3a-8f15-832f2f7eb8bd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:49.558161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:49.558847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.563958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.564275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] PATH_INFO: `/v3/projects/d33286ef83df49bab1df974e8f01f8cc/users/61be0b8995594c6c8d79da0df6325f89/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.594511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d33286ef83df49bab1df974e8f01f8cc, user_id=61be0b8995594c6c8d79da0df6325f89, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:49.598213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:49.616953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1260/2513] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:49 2022] PUT /identity/v3/projects/d33286ef83df49bab1df974e8f01f8cc/users/61be0b8995594c6c8d79da0df6325f89/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:49.622660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:49.639479 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2c184f-31d5-4872-a456-58b498327433 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:49.719996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ba6cf0-d9e0-4081-a412-04393ba19d9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:49.720303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba6cf0-d9e0-4081-a412-04393ba19d9f admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.720512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba6cf0-d9e0-4081-a412-04393ba19d9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.720784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ba6cf0-d9e0-4081-a412-04393ba19d9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.721461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ba6cf0-d9e0-4081-a412-04393ba19d9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:49.723207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ba6cf0-d9e0-4081-a412-04393ba19d9f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:49.740451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:49.740631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1254/2514] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:49.754737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1251db7f-5de2-4608-b655-875d00960899 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:49.764758 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:49.765498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:49.765878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:49.766284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] PATH_INFO: `/v3/projects/d33286ef83df49bab1df974e8f01f8cc/users/61be0b8995594c6c8d79da0df6325f89/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:49.770116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=d33286ef83df49bab1df974e8f01f8cc, user_id=61be0b8995594c6c8d79da0df6325f89, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:49.773295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:49.784437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1261/2515] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:49 2022] PUT /identity/v3/projects/d33286ef83df49bab1df974e8f01f8cc/users/61be0b8995594c6c8d79da0df6325f89/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:54.693134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46cd4af-d031-425a-934b-8fe4ef110b3f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.693680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46cd4af-d031-425a-934b-8fe4ef110b3f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.693776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46cd4af-d031-425a-934b-8fe4ef110b3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.731265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f46cd4af-d031-425a-934b-8fe4ef110b3f None None] MFA Rules not processed for user `61be0b8995594c6c8d79da0df6325f89`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:54.732414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f46cd4af-d031-425a-934b-8fe4ef110b3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:54.812904 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1255/2516] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:14:54 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:54.827041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.829056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:54.896825 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82029dac-998a-45a7-945c-fa959a7961cf admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.900793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.901645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.902096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.902372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.932923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:54.972659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.974253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.984604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1262/2517] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:54 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:55.161594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cd15bf-53bd-4d73-b366-2af7fcae5a2c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.161879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cd15bf-53bd-4d73-b366-2af7fcae5a2c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.162148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cd15bf-53bd-4d73-b366-2af7fcae5a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.182594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-51cd15bf-53bd-4d73-b366-2af7fcae5a2c None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:55.183838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-51cd15bf-53bd-4d73-b366-2af7fcae5a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:55.190333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:55.243473 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1256/2518] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:55.255442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.256894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:55.360464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe84db64-36e0-4af1-9c87-526602fb7046 service glance] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.365919 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d208c7-1ebb-4d55-9ca4-fe3ad0b10af4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.366538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d208c7-1ebb-4d55-9ca4-fe3ad0b10af4 service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.366860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d208c7-1ebb-4d55-9ca4-fe3ad0b10af4 service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.367132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d208c7-1ebb-4d55-9ca4-fe3ad0b10af4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.377507 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d208c7-1ebb-4d55-9ca4-fe3ad0b10af4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.379140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d208c7-1ebb-4d55-9ca4-fe3ad0b10af4 service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.388143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1263/2519] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:15:04.971120 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:04.971715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:04.971990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:04.992414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] MFA Rules not processed for user `75c0e91cec2b455f82574ed9f110be92`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:04.993960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:05.005797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1257/2520] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:15:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:05.384508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=48 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:07.386309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:07.387256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:07.387696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:07.408914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:07.410081 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:07.418292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1264/2521] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:15:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:07.445913 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:07.461089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f21dee9-d43e-4d76-a271-acfde146d97f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:07.468198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d89635-bf56-4246-a26a-a251b09109db service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:07.469042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d89635-bf56-4246-a26a-a251b09109db service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:07.469413 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d89635-bf56-4246-a26a-a251b09109db service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:07.470004 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d89635-bf56-4246-a26a-a251b09109db service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:07.478852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d89635-bf56-4246-a26a-a251b09109db service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:07.480983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d89635-bf56-4246-a26a-a251b09109db service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:07.493607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1258/2522] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:15:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:15:17.402190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:17.488403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=29 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:20.057240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:20.058667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:20.114108 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32515ee1-ea24-427b-b411-86e9147d5e52 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:20.117707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:20.118313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:20.118595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:20.118872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:20.120655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:20.206159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:20.209238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:20.219298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1265/2523] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:20 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:30.214584 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=50 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:34.101551 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ac086-80ee-45fb-9244-c0491c8ef20f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:34.102067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ac086-80ee-45fb-9244-c0491c8ef20f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:34.102196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ac086-80ee-45fb-9244-c0491c8ef20f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:34.121484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2f1ac086-80ee-45fb-9244-c0491c8ef20f None None] MFA Rules not processed for user `f16c1cde815c4a2580cd5ebf7b3ed76d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:34.122672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2f1ac086-80ee-45fb-9244-c0491c8ef20f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:34.177063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1259/2524] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:15:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:42.692819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:42.701470 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d3904e-4fde-4722-9595-cbb9dacd0131 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:42.705097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:42.705698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:42.705973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:42.706240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:42.715429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:42.716916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:42.726678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1266/2525] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:42 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:43.952687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:43.953308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:43.953409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:43.972023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] MFA Rules not processed for user `53c020af94d64bac81df7837a86586b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:43.973372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:44.028739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1260/2526] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:15:43 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:52.722371 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:54.022068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=50 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:18.641834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:18.654559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb68282-6edb-4615-83cb-1aef570e2901 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:18.659525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:18.660467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:18.661276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:18.661773 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:18.664027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:18.746694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:18.748194 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd7ea78-1e47-463c-a0d2-d4276f9f87f7 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:18.750252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:18.759961 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1267/2527] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:18.775519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524aef05-45d8-49dd-b42a-0e4ffabb55ce None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:18.779386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:18.780042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:18.780323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:18.780595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:18.783003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:18.798626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:18.800097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67d165-db4b-40e3-ab20-8546749b5809 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:18.817900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1261/2528] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:16:18.961819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:18.962364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:18.963508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:18.987249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] MFA Rules not processed for user `066c7318255e4a01a0f6a81715718e15`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:18.988495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:19.043732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1268/2529] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:16:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:22.758691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:22.759240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:22.759240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:22.779656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] MFA Rules not processed for user `b38384d843d04503b69e5cb61af6c6ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:22.780908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:22.842341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1262/2530] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:16:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:16:25.624753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:25.626980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:25.650165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2173f10d-4636-47bb-bcf3-e4f2b6af9f45 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:25.658200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:25.658827 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:25.659109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:25.659381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:25.661207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:25.676579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:25.678398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:25.688782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1269/2531] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:16:32.834915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=31 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:35.683756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=65 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:36.178819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:36.180204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:36.235280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97597f68-6eba-4f20-bfc5-5898ec4254be None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:36.239898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:36.240502 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:36.240787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:36.241089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:36.242807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:36.254522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:36.255989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:36.265518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1263/2532] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:36.297013 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:36.299073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:36.314849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88f9f7d-26aa-4dcc-b8e3-f7f6e4fd008f service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:36.320997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:36.321677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:36.322026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:36.322364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:36.324164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:36.336913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:36.338632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:36.348509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1270/2533] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:36 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:42.039986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:42.049422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3a53ee-7f94-4d56-8011-ce35bb330283 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:42.053470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:42.054059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:42.054338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:42.054607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:42.056322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:42.066521 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:42.068022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:42.075717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1264/2534] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:46.343664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:52.072307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=32 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:52.199513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:52.211268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62daa9bb-4624-4c24-a9d0-346ca2f8660a service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:52.216774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:52.217341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:52.217623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:52.217898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:52.219598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:52.231313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:52.232892 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f677e4b-a8e6-41f2-bec7-0a337e7585a6 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:52.243931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1271/2535] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:55.005971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:55.014578 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87b532f-12a1-4279-ab46-800e39572622 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:55.018234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:55.019458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:55.019851 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:55.020130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:55.021865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:55.087463 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:55.089339 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:55.100474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1265/2536] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:57.492474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:57.495057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:57.495511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:57.515272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] MFA Rules not processed for user `f16c1cde815c4a2580cd5ebf7b3ed76d`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:57.516515 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:57.524768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1272/2537] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:16:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:58.043690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:58.052256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5fc895-eccd-45d2-a937-31dfda7324e8 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:58.057218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:58.057975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:58.058310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:58.058650 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:58.060459 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:58.071506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:58.072936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce68658d-9689-447d-9353-60cef957af79 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:58.081760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1266/2538] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:58.471557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:58.473154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:58.585635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f426ddb-4698-43f0-b17a-778e4e1d8ef7 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:58.594186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:58.595041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:58.595324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:58.595602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:58.598130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:58.614293 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:58.615995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632eb75b-6a5b-4b25-9f6e-9156bb2f0837 service placement] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:58.626692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1273/2539] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:58.794249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853c3295-ceab-4c4c-b9e3-ca1f5151fdaf None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:58.794580 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853c3295-ceab-4c4c-b9e3-ca1f5151fdaf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:58.795047 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853c3295-ceab-4c4c-b9e3-ca1f5151fdaf None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:58.796838 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1267/2540] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:16:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:16:58.808741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:58.809100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:58.809383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:58.836770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] MFA Rules not processed for user `f929f94af045445bb0a608dee02a7c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:58.838215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:58.847852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1274/2541] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:16:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:58.864329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853c3295-ceab-4c4c-b9e3-ca1f5151fdaf None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:58.868220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-853c3295-ceab-4c4c-b9e3-ca1f5151fdaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:58.883876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:58.886035 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:58.894581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853c3295-ceab-4c4c-b9e3-ca1f5151fdaf None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:58.901299 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:58.902012 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:58.902358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:58.902692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:58.907822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856bee29-77ad-46cf-8621-2548bc9acb76 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:58.913771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:58.914464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:58.914651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:58.914903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:58.915174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] PATH_INFO: `/v3/users/53c020af94d64bac81df7837a86586b2` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:58.916307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53c020af94d64bac81df7837a86586b2)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:58.916729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:58.918389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:58.928849 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1268/2542] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:58.981428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53c020af94d64bac81df7837a86586b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:58.995674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53c020af94d64bac81df7837a86586b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.003586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53c020af94d64bac81df7837a86586b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.005029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53c020af94d64bac81df7837a86586b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.006176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1733b7f9-51dc-4415-946c-5b61e83fe722 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53c020af94d64bac81df7837a86586b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.021602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1275/2543] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:16:58 2022] DELETE /identity/v3/users/53c020af94d64bac81df7837a86586b2 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:16:59.195280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.207130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10487277-980b-483b-b241-54dcd4f8f75e service placement] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.269491 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.270039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.270322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.270597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] PATH_INFO: `/v3/projects/320dd138f254473a84a601c153e1e39b` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.271740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] RBAC: Authorizing `identity:delete_project(project_id=320dd138f254473a84a601c153e1e39b)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.273727 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.324928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '320dd138f254473a84a601c153e1e39b', 'request_id': 'req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.332523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '320dd138f254473a84a601c153e1e39b', 'request_id': 'req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.338784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '320dd138f254473a84a601c153e1e39b', 'request_id': 'req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.345298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:59.351226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] The token cache is being invalidate because project 320dd138f254473a84a601c153e1e39b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:16:59.351757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.352346 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-840df4c2-5b13-4f0b-9346-d0fbf8b4d750 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:59.354362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1269/2544] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:16:59 2022] DELETE /identity/v3/projects/320dd138f254473a84a601c153e1e39b => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:59.359339 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.359634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.359914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.380182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:59.381484 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:59.421203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:59.474223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1276/2545] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:16:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:59.476853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.477326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.477668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.498500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:59.499717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:59.508106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1270/2546] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:16:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:59.512349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.522655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55085aec-4cd1-40fa-9ab0-e1cfafcea416 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.527080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.527670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.527943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.528209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.528842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:16:59.529169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.530741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.542459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1277/2547] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:16:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:16:59.546684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.555837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b87b19-2e7b-4b08-9caa-6d26b647de27 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.560417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.561105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.561383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.561651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.562269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.564137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.594998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1271/2548] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:16:59 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:16:59.599707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.610091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421c0e7-ad5e-42b6-a520-0615f56ee85c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.670340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.670943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.671219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.671492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.672130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.674084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.696260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1278/2549] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:16:59 2022] POST /identity/v3/users => generated 294 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:16:59.700245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.711064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d2bec9-5ae0-4b63-89d2-43154a025094 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.715684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.716309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.716594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.716883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.717531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.719110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.728881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1272/2550] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:16:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:16:59.732701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.743625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b677ca97-0dc1-4e0d-a02b-fd10254edcea admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.748210 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf261b3-1817-4421-812a-f1e8d207b97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.748800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf261b3-1817-4421-812a-f1e8d207b97d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.749125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf261b3-1817-4421-812a-f1e8d207b97d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.749403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf261b3-1817-4421-812a-f1e8d207b97d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.750052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf261b3-1817-4421-812a-f1e8d207b97d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.751677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf261b3-1817-4421-812a-f1e8d207b97d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.763012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1279/2551] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:16:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:16:59.767092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.777753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b99d4f-461e-4b5e-9405-dd08469f8114 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.782341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.782986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.783261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.783527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] PATH_INFO: `/v3/projects/0a769dbf20fa4526a8c63c4b631c94b1/users/e077cbccb35940608bc2224c373c0b99/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.798721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a769dbf20fa4526a8c63c4b631c94b1, user_id=e077cbccb35940608bc2224c373c0b99, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:59.802086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:59.816008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1273/2552] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:16:59 2022] PUT /identity/v3/projects/0a769dbf20fa4526a8c63c4b631c94b1/users/e077cbccb35940608bc2224c373c0b99/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:59.821271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ac647-68cb-4eb0-8fba-9085fc09131e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.821624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ac647-68cb-4eb0-8fba-9085fc09131e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.821987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ac647-68cb-4eb0-8fba-9085fc09131e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:59.840846 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3d1ac647-68cb-4eb0-8fba-9085fc09131e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:59.842176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3d1ac647-68cb-4eb0-8fba-9085fc09131e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:59.907301 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1280/2553] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:16:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:59.917743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:59.919523 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:59.983671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1faad-0094-4ad8-88bd-2f9f0bd54163 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:59.988621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:59.989281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:59.989565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:59.989841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:00.000189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:00.001678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:00.021097 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1274/2554] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:04.679565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42618728-ddf8-446d-a479-b9de2bea0bb4 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:04.680011 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42618728-ddf8-446d-a479-b9de2bea0bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:04.680119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42618728-ddf8-446d-a479-b9de2bea0bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:04.712025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-42618728-ddf8-446d-a479-b9de2bea0bb4 None None] MFA Rules not processed for user `e077cbccb35940608bc2224c373c0b99`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:04.713365 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-42618728-ddf8-446d-a479-b9de2bea0bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:04.744636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:04.785057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1281/2555] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:04.797118 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:04.800840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:04.891202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3601bd-6ee7-4b81-983f-0a8f2b02c9a8 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:04.897453 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:04.898227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:04.898577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:04.898918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:04.907922 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:04.909607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:04.919584 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1275/2556] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:04 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:06.299653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:06.300138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:06.300234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:06.332738 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] MFA Rules not processed for user `e077cbccb35940608bc2224c373c0b99`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:06.334030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:06.340938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1282/2557] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:06 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:06.656773 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:06.667954 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3fe0d-42b8-4696-8379-1c103596eb38 service nova] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:06.671318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:06.671918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:06.672203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:06.672495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:06.674743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:06.685106 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:06.686248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:06.695642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1276/2558] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:07.755414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:07.774499 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a0a8b-a4d5-4045-9675-03be758c523a None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:07.783026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:07.784917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:07.786774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:07.787503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:07.790561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:07.807514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:07.809460 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021cee45-edda-46b5-b8cb-191235c0c544 service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:07.822834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1283/2559] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:11.878264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:11.892843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7da8b5-9f9d-4062-9792-5e2992bda07a service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:11.898417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:11.899043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:11.899318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:11.899587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:11.901322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:11.911207 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:11.913832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:11.915981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:11.927992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1277/2560] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:17.817090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=22 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:21.923215 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:32.108127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:32.108604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:32.108707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:32.127856 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:32.129064 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:32.183263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1284/2561] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 29 02:17:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:32.223808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:32.225242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:32.293383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5ee60a-4a7f-49ed-9e3d-aeb4013ee167 service nova] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:32.298261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:32.298854 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:32.299133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:32.299401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:32.309745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:32.311316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:32.319995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1278/2562] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:17:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:17:39.816343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:39.818020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:39.833226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a890c9b-55b0-4e85-9ffd-1e72cbe85687 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:39.841667 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:39.842821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:39.843271 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:39.843671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] PATH_INFO: `/v3/users/75c0e91cec2b455f82574ed9f110be92` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:39.844910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=75c0e91cec2b455f82574ed9f110be92)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:39.847056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:39.895536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75c0e91cec2b455f82574ed9f110be92'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:39.911523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75c0e91cec2b455f82574ed9f110be92'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:39.921433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75c0e91cec2b455f82574ed9f110be92'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:39.922246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75c0e91cec2b455f82574ed9f110be92'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:39.923296 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75c0e91cec2b455f82574ed9f110be92'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:39.945637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1285/2563] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:17:39 2022] DELETE /identity/v3/users/75c0e91cec2b455f82574ed9f110be92 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:40.451298 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:40.465737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba81fcd-b897-4b4b-8014-a27181286893 service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:40.554030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:40.555077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:40.555733 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:40.556685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] PATH_INFO: `/v3/projects/40f01307928c46918e93db66b6473733` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:40.559160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40f01307928c46918e93db66b6473733)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:40.561387 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:40.628532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40f01307928c46918e93db66b6473733', 'request_id': 'req-8684e352-3598-460e-9681-c38d32e42100'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.649638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40f01307928c46918e93db66b6473733', 'request_id': 'req-8684e352-3598-460e-9681-c38d32e42100'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.659697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40f01307928c46918e93db66b6473733', 'request_id': 'req-8684e352-3598-460e-9681-c38d32e42100'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.673115 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] The token cache is being invalidate because project 40f01307928c46918e93db66b6473733 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:17:40.673661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.674978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.680953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1279/2564] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:17:40 2022] DELETE /identity/v3/projects/40f01307928c46918e93db66b6473733 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:40.681693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:40.683242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:40.704735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbeff2c-8153-4965-b8da-272739d57f7d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:40.835182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:40.835911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:40.836376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:40.836691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] PATH_INFO: `/v3/users/61be0b8995594c6c8d79da0df6325f89` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:40.838275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=61be0b8995594c6c8d79da0df6325f89)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:40.841948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:40.866080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:40.901899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61be0b8995594c6c8d79da0df6325f89'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.917173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61be0b8995594c6c8d79da0df6325f89'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.932063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61be0b8995594c6c8d79da0df6325f89'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.933413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61be0b8995594c6c8d79da0df6325f89'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.934761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c768ddf4-d798-4e81-abc2-c46687129a6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61be0b8995594c6c8d79da0df6325f89'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:40.955222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1286/2565] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:17:40 2022] DELETE /identity/v3/users/61be0b8995594c6c8d79da0df6325f89 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:17:41.140458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:41.142091 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:41.156311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684e352-3598-460e-9681-c38d32e42100 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:41.179319 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:41.230062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:41.230878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:41.231292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:41.231677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] PATH_INFO: `/v3/projects/d33286ef83df49bab1df974e8f01f8cc` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:41.233135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d33286ef83df49bab1df974e8f01f8cc)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:41.235439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:41.300764 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd33286ef83df49bab1df974e8f01f8cc', 'request_id': 'req-a770f477-0600-4100-a108-8c58ac750422'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:41.315569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd33286ef83df49bab1df974e8f01f8cc', 'request_id': 'req-a770f477-0600-4100-a108-8c58ac750422'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:41.324141 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd33286ef83df49bab1df974e8f01f8cc', 'request_id': 'req-a770f477-0600-4100-a108-8c58ac750422'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:41.339001 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] The token cache is being invalidate because project d33286ef83df49bab1df974e8f01f8cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:17:41.339296 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:41.340117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-a770f477-0600-4100-a108-8c58ac750422 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:41.342157 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1280/2566] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:17:41 2022] DELETE /identity/v3/projects/d33286ef83df49bab1df974e8f01f8cc => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:44.469302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f19f82-a945-498d-9a32-03a814d2ab7d None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.469876 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f19f82-a945-498d-9a32-03a814d2ab7d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.470019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f19f82-a945-498d-9a32-03a814d2ab7d None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.471998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1287/2567] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:17:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:17:44.478136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41086fa6-38fe-4da6-b145-54239eb1712a None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.478588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41086fa6-38fe-4da6-b145-54239eb1712a None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.478888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41086fa6-38fe-4da6-b145-54239eb1712a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.501887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-41086fa6-38fe-4da6-b145-54239eb1712a None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:44.503124 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-41086fa6-38fe-4da6-b145-54239eb1712a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:44.554576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1281/2568] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:17:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:44.567234 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f19f82-a945-498d-9a32-03a814d2ab7d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:44.569076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-29f19f82-a945-498d-9a32-03a814d2ab7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:44.583150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f19f82-a945-498d-9a32-03a814d2ab7d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:44.587431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741b8cfb-bc38-4fed-88b0-3e98555d3d06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:44.588044 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741b8cfb-bc38-4fed-88b0-3e98555d3d06 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.588326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741b8cfb-bc38-4fed-88b0-3e98555d3d06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.589168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741b8cfb-bc38-4fed-88b0-3e98555d3d06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.596410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741b8cfb-bc38-4fed-88b0-3e98555d3d06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:44.597952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741b8cfb-bc38-4fed-88b0-3e98555d3d06 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:44.606166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1288/2569] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:44.687900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62630ced-0e3e-4e6c-b927-39ee250e3b70 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.688184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62630ced-0e3e-4e6c-b927-39ee250e3b70 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.688458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62630ced-0e3e-4e6c-b927-39ee250e3b70 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.690694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1282/2570] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:17:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:17:44.695845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.696130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.696404 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.715720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:44.717024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:44.789161 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1289/2571] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:17:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:44.799691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62630ced-0e3e-4e6c-b927-39ee250e3b70 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:44.810517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62630ced-0e3e-4e6c-b927-39ee250e3b70 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:44.814016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:44.815347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.815630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.815903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.826033 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:44.827486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:44.838543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1283/2572] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:47.710280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:47.711728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:47.723355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1c8d89-15e2-487d-8d2f-18fa69864d09 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:47.728085 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:47.728940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:47.729218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:47.729656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:47.731723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:47.794364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:47.795809 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:47.805537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1290/2573] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:51.090823 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.092465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:51.107389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e37a3f6-ae69-4c8c-833f-a91e3ff9f934 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.112436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.113089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.113396 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.113667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] PATH_INFO: `/v3/users/066c7318255e4a01a0f6a81715718e15` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.114784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=066c7318255e4a01a0f6a81715718e15)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.116692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.157799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:51.166247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '066c7318255e4a01a0f6a81715718e15'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.173267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '066c7318255e4a01a0f6a81715718e15'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.173943 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '066c7318255e4a01a0f6a81715718e15'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.175080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '066c7318255e4a01a0f6a81715718e15'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.182380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-404c58bc-f679-40dc-a9d0-f16947a467d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '066c7318255e4a01a0f6a81715718e15'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.200278 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1284/2574] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:17:51 2022] DELETE /identity/v3/users/066c7318255e4a01a0f6a81715718e15 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:51.376545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.388230 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97f0379-07ec-49b8-8c56-6b1b69fd38e9 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.393436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:51.451381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.452022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.452302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.452580 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] PATH_INFO: `/v3/projects/2a04f2488c2a4d219ad8193c4dd12b65` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.453772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a04f2488c2a4d219ad8193c4dd12b65)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.455771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.514879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a04f2488c2a4d219ad8193c4dd12b65', 'request_id': 'req-e13d58a6-0b70-414e-b63c-b8531dbcb097'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.522491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a04f2488c2a4d219ad8193c4dd12b65', 'request_id': 'req-e13d58a6-0b70-414e-b63c-b8531dbcb097'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.528464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a04f2488c2a4d219ad8193c4dd12b65', 'request_id': 'req-e13d58a6-0b70-414e-b63c-b8531dbcb097'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.542918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] The token cache is being invalidate because project 2a04f2488c2a4d219ad8193c4dd12b65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:17:51.543209 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.543857 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-e13d58a6-0b70-414e-b63c-b8531dbcb097 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:51.545396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1291/2575] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:17:51 2022] DELETE /identity/v3/projects/2a04f2488c2a4d219ad8193c4dd12b65 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:51.550860 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.551139 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.551499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.573132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:51.574388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:51.632942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1285/2576] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:17:51.641399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.641681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.641954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.663767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:51.665116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:51.672837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1292/2577] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:51.677852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.687429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113ced01-a029-4379-a5b9-3937e8ff2102 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.692349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.692994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.693268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.693545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.694190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:17:51.694495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.696080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.707619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1286/2578] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:17:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:17:51.711708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.721516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7903767b-0fff-4841-9591-57a95c6f65fe None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.726101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.726705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.726982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.727248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.727897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.729820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.761897 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1293/2579] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:17:51 2022] POST /identity/v3/projects => generated 403 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:17:51.763082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.773048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c7354d-3fae-4544-bbf1-c38acdb5a47b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.832010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.832619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.832930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.833211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.833845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.835794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.857721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1287/2580] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:17:51 2022] POST /identity/v3/users => generated 302 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:17:51.861737 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.876819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12deb41c-c1a8-44c8-92b8-1d415d3e6369 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.881666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.882266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.882544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.882815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.883513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.885124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.896024 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1294/2581] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:51.900018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.910572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3269bfd0-d01d-4abd-9558-3bbed6861c3b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.915184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b32c54f-2020-46b2-86e3-b77c0572e6ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.915784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b32c54f-2020-46b2-86e3-b77c0572e6ff admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.916067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b32c54f-2020-46b2-86e3-b77c0572e6ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.916340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b32c54f-2020-46b2-86e3-b77c0572e6ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.917036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b32c54f-2020-46b2-86e3-b77c0572e6ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.918688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b32c54f-2020-46b2-86e3-b77c0572e6ff admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.929804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1288/2582] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:51.933544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:51.943660 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2425a8c8-0eba-4994-b3f4-72775040a78f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:51.948428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:51.949114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.949397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.949677 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] PATH_INFO: `/v3/projects/ba5b84d921174e7582b824a1400c0d1e/users/cad4c337d5af45d3a160e303f348f904/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:51.964342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba5b84d921174e7582b824a1400c0d1e, user_id=cad4c337d5af45d3a160e303f348f904, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:51.967020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:51.980099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1295/2583] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:17:51 2022] PUT /identity/v3/projects/ba5b84d921174e7582b824a1400c0d1e/users/cad4c337d5af45d3a160e303f348f904/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:51.985224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0051fcd-5152-4641-9e46-9fcdb3dca6bf None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:51.985515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0051fcd-5152-4641-9e46-9fcdb3dca6bf None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:51.985796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0051fcd-5152-4641-9e46-9fcdb3dca6bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:52.004563 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f0051fcd-5152-4641-9e46-9fcdb3dca6bf None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:52.005840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f0051fcd-5152-4641-9e46-9fcdb3dca6bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:52.024292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:52.066747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1289/2584] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:52.076259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:52.077981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:52.117322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:52.135536 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e538af-62ee-40dc-befe-c1ded791ad92 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:52.139045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:52.139637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:52.139912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:52.140180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:52.149541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:52.150959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:52.160592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1296/2585] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:52.774507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:52.775008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:52.775134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:52.804297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] MFA Rules not processed for user `cad4c337d5af45d3a160e303f348f904`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:52.805577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:52.874254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1290/2586] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:52 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:17:52.878970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:52.889691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc2f392-ad5f-43d6-a21f-38e2908e63bc service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:52.895583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:52.895866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:52.896143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:52.896439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:52.897219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:52.899051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:52.931689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1297/2587] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:17:52 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:17:52.933148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:52.947625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e029a9e-d78b-4c5a-a314-6a2d497cb38c None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:53.023121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:53.024029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:53.024382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:53.024721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:53.025442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:53.027420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:53.053029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1291/2588] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:17:52 2022] POST /identity/v3/users => generated 309 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:17:53.055805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:53.067534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa3a503-6655-46e6-9ebb-0ab2a3538fb0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:53.076246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:53.078576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:53.078576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:53.078576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:53.078576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:53.080439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:53.092547 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1298/2589] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:53.097581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:53.107827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a945b3a9-327d-42b0-b643-9b314628e4a0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:53.113105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:53.113751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:53.114030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:53.114302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] PATH_INFO: `/v3/projects/986f34cff3dc4279826f00b5cfc7c706/users/934d20a7bfee4d00894ddf97d46d306f/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:53.129697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] RBAC: Authorizing `identity:create_grant(project_id=986f34cff3dc4279826f00b5cfc7c706, user_id=934d20a7bfee4d00894ddf97d46d306f, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:53.132305 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:53.145166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1292/2590] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:17:53 2022] PUT /identity/v3/projects/986f34cff3dc4279826f00b5cfc7c706/users/934d20a7bfee4d00894ddf97d46d306f/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:17:53.149101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:53.159349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5c60d6-477d-481b-8d5e-48149dbc51e0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:53.213904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487d318e-6477-4263-9e46-b05929624578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:53.214502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487d318e-6477-4263-9e46-b05929624578 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:53.214778 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487d318e-6477-4263-9e46-b05929624578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:53.215052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487d318e-6477-4263-9e46-b05929624578 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:53.215674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487d318e-6477-4263-9e46-b05929624578 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:53.217291 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487d318e-6477-4263-9e46-b05929624578 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:53.227099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1299/2591] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:53 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:53.231218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:53.242217 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35df230-056c-4008-8841-918b4645f591 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:53.246749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:53.247390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:53.247665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:53.247933 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] PATH_INFO: `/v3/projects/986f34cff3dc4279826f00b5cfc7c706/users/934d20a7bfee4d00894ddf97d46d306f/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:53.249847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=986f34cff3dc4279826f00b5cfc7c706, user_id=934d20a7bfee4d00894ddf97d46d306f, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:53.252448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:53.263008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1293/2592] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:17:53 2022] PUT /identity/v3/projects/986f34cff3dc4279826f00b5cfc7c706/users/934d20a7bfee4d00894ddf97d46d306f/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:53.884039 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:53.884629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:53.884748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:53.914904 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] MFA Rules not processed for user `934d20a7bfee4d00894ddf97d46d306f`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:53.916176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:53.987419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1300/2593] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:53 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:17:54.484962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:54.487604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:54.498589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:54.579982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b26a9e-6738-4def-b856-21cba28d26f4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:54.586515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:54.587138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:54.587421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:54.588204 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:54.590160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:54.606493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:54.609636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:54.621423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1294/2594] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:55.050214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:55.081173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:55.114900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b6c3b6-50f9-41e3-9644-29dc30c677b8 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:55.119411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:55.120032 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:55.120306 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:55.120577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] PATH_INFO: `/v3/users/cad4c337d5af45d3a160e303f348f904` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:55.121850 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cad4c337d5af45d3a160e303f348f904)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:55.123770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:55.167982 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cad4c337d5af45d3a160e303f348f904'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.180781 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cad4c337d5af45d3a160e303f348f904'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.188256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cad4c337d5af45d3a160e303f348f904'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.189086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cad4c337d5af45d3a160e303f348f904'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.190525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cad4c337d5af45d3a160e303f348f904'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.203501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1301/2595] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:17:55 2022] DELETE /identity/v3/users/cad4c337d5af45d3a160e303f348f904 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:55.380309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:55.398270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b395ef16-f859-4533-9002-1f97f8479ae2 service placement] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:55.457982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:55.458640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:55.458921 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:55.459193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] PATH_INFO: `/v3/projects/ba5b84d921174e7582b824a1400c0d1e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:55.460311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba5b84d921174e7582b824a1400c0d1e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:55.462230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:55.510537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba5b84d921174e7582b824a1400c0d1e', 'request_id': 'req-6ecef200-95d4-41a8-9926-5343a1fbf879'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.517617 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba5b84d921174e7582b824a1400c0d1e', 'request_id': 'req-6ecef200-95d4-41a8-9926-5343a1fbf879'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.524168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba5b84d921174e7582b824a1400c0d1e', 'request_id': 'req-6ecef200-95d4-41a8-9926-5343a1fbf879'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.534256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] The token cache is being invalidate because project ba5b84d921174e7582b824a1400c0d1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:17:55.534544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.535163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.536624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1295/2596] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:17:55 2022] DELETE /identity/v3/projects/ba5b84d921174e7582b824a1400c0d1e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:55.540668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:55.542123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:55.555335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67bf0954-85df-4bd5-b72e-df9689f5d6b8 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:55.611832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:55.612479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:55.612761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:55.613065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] PATH_INFO: `/v3/users/934d20a7bfee4d00894ddf97d46d306f` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:55.614197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=934d20a7bfee4d00894ddf97d46d306f)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:55.616182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:55.666036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '934d20a7bfee4d00894ddf97d46d306f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.667058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:55.685633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '934d20a7bfee4d00894ddf97d46d306f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.693572 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '934d20a7bfee4d00894ddf97d46d306f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.694289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '934d20a7bfee4d00894ddf97d46d306f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.694896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82f99951-affe-4a85-ab5e-1ddb701497c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '934d20a7bfee4d00894ddf97d46d306f'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:55.708169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1302/2597] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:17:55 2022] DELETE /identity/v3/users/934d20a7bfee4d00894ddf97d46d306f => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:55.882089 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:55.884135 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:55.899198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecef200-95d4-41a8-9926-5343a1fbf879 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:55.930458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:55.971831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:55.972676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:55.973070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:55.973986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] PATH_INFO: `/v3/projects/986f34cff3dc4279826f00b5cfc7c706` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:55.975417 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] RBAC: Authorizing `identity:delete_project(project_id=986f34cff3dc4279826f00b5cfc7c706)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:55.977616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.029564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '986f34cff3dc4279826f00b5cfc7c706', 'request_id': 'req-5218a64d-21b9-4f16-bb05-8caf1b203550'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:56.037344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '986f34cff3dc4279826f00b5cfc7c706', 'request_id': 'req-5218a64d-21b9-4f16-bb05-8caf1b203550'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:56.043825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '986f34cff3dc4279826f00b5cfc7c706', 'request_id': 'req-5218a64d-21b9-4f16-bb05-8caf1b203550'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:56.055426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] The token cache is being invalidate because project 986f34cff3dc4279826f00b5cfc7c706 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:17:56.055728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:56.056358 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-5218a64d-21b9-4f16-bb05-8caf1b203550 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:17:56.058166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1296/2598] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:17:55 2022] DELETE /identity/v3/projects/986f34cff3dc4279826f00b5cfc7c706 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:56.063027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.063568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.063681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.088798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:56.090114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:56.153855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1303/2599] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:17:56.163471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.163752 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.164031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.183807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:56.185065 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:56.193938 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1297/2600] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:56.198423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.209066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44886e-b27d-4bf5-8d9a-a1d4c7c7b656 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.213621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.214219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.214497 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.214767 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.215432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:17:56.215731 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.217361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.228768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1304/2601] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:17:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:17:56.233206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.242422 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b10a49-35d6-42bc-9a2a-c8d32401f11c None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.247086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.247686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.247989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.248262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.248924 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.250830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.278075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1298/2602] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/projects => generated 363 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:17:56.282486 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.295235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfe30d-7fdd-4e4d-acb2-2c352cefa9c6 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.350124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.350730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.351009 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.351281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.351913 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.353866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.381516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1305/2603] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/users => generated 282 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:17:56.385624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.396591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c478ce-53d4-44da-98f1-84005618b582 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.401351 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.401949 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.402231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.402504 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.403140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.404750 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.415145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1299/2604] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:56.418975 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.428937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab83866-d931-4d55-8ea7-dd3328753dc7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.433785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78acfd0-9a93-44e0-bd1a-daf936eeab67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.434312 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78acfd0-9a93-44e0-bd1a-daf936eeab67 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.434592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78acfd0-9a93-44e0-bd1a-daf936eeab67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.434866 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78acfd0-9a93-44e0-bd1a-daf936eeab67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.435506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78acfd0-9a93-44e0-bd1a-daf936eeab67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.437125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78acfd0-9a93-44e0-bd1a-daf936eeab67 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.447708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1306/2605] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:56.451666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.462078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2180923-9bfb-4429-8412-2dd9f195d044 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.466700 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.467338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.467623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.467901 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] PATH_INFO: `/v3/projects/ad2e16c79a244de9a8a73a55be7de22a/users/a243271ff1b54fa690c506beb080061c/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.482683 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad2e16c79a244de9a8a73a55be7de22a, user_id=a243271ff1b54fa690c506beb080061c, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.485368 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.499609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1300/2606] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:17:56 2022] PUT /identity/v3/projects/ad2e16c79a244de9a8a73a55be7de22a/users/a243271ff1b54fa690c506beb080061c/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:56.509926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.510235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.510509 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.538692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] MFA Rules not processed for user `a243271ff1b54fa690c506beb080061c`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:56.539898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:56.558022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:56.621504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1307/2607] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:17:56.625669 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.682209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:56.694734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e1aa4d-9a8e-4e2a-974d-05a88dede398 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.700280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.700990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.701274 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.701548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.702188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.704067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.731272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1301/2608] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/projects => generated 365 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:17:56.735976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.747191 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2b2bd-6fe9-4c84-869d-2114dad5f8e3 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.802402 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.803000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.803282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.803565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.804285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.806228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.827190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1308/2609] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:17:56 2022] POST /identity/v3/users => generated 289 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:17:56.831420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.841636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f9b62f-328f-45bd-b88f-b211ba51b747 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.846785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.847379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.847664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.847936 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.848571 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.850210 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.860466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1302/2610] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:56.864994 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.874952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57fc308-2377-426c-bb09-c2c8b335ff23 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.879766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.880412 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.880697 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.881006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] PATH_INFO: `/v3/projects/651575866eb04cab96ea93baeb65f0c4/users/86b75ed88910450e9d78392c3332cf68/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.896005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] RBAC: Authorizing `identity:create_grant(project_id=651575866eb04cab96ea93baeb65f0c4, user_id=86b75ed88910450e9d78392c3332cf68, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.898949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.911863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1309/2611] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:17:56 2022] PUT /identity/v3/projects/651575866eb04cab96ea93baeb65f0c4/users/86b75ed88910450e9d78392c3332cf68/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:17:56.915825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:56.925688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d58de5a-58ed-4636-9b42-c4a55ee7048d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:56.981442 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a18f8b7-914b-44f3-af17-3061ab481b40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:56.981967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18f8b7-914b-44f3-af17-3061ab481b40 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:56.982246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18f8b7-914b-44f3-af17-3061ab481b40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:56.982517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18f8b7-914b-44f3-af17-3061ab481b40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:56.983144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18f8b7-914b-44f3-af17-3061ab481b40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:56.984723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18f8b7-914b-44f3-af17-3061ab481b40 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:56.994332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1303/2612] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:17:56 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:17:56.998640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:57.009405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e606e93-d122-41ac-8e20-66ffb83c769f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:57.014021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:57.014663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:57.014943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:57.015214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] PATH_INFO: `/v3/projects/651575866eb04cab96ea93baeb65f0c4/users/86b75ed88910450e9d78392c3332cf68/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:57.017167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] RBAC: Authorizing `identity:create_grant(project_id=651575866eb04cab96ea93baeb65f0c4, user_id=86b75ed88910450e9d78392c3332cf68, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:57.020341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:57.033356 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1310/2613] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:17:56 2022] PUT /identity/v3/projects/651575866eb04cab96ea93baeb65f0c4/users/86b75ed88910450e9d78392c3332cf68/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:17:57.043505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:57.043793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:57.044068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:57.074963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] MFA Rules not processed for user `86b75ed88910450e9d78392c3332cf68`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:57.076167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:57.141895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1304/2614] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:17:57 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:57.153852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:57.155635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:57.212638 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0cd4e3-01b4-4f02-b94c-32ecb556a05d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:57.216416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:57.217049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:57.217326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:57.217594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:57.226872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:57.228297 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:57.237553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1311/2615] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:57 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:57.313991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:57.323973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e076ff47-3835-44f6-a909-966591a69e3e None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:57.327600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02bd1ae-52a3-4dd7-9e0c-5a3a99066228 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:57.328205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02bd1ae-52a3-4dd7-9e0c-5a3a99066228 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:57.328505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02bd1ae-52a3-4dd7-9e0c-5a3a99066228 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:57.328780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02bd1ae-52a3-4dd7-9e0c-5a3a99066228 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:57.375595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:57.390066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02bd1ae-52a3-4dd7-9e0c-5a3a99066228 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:57.391541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02bd1ae-52a3-4dd7-9e0c-5a3a99066228 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:57.401493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1305/2616] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:57 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:07.233378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=88,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:07.397009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=6 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:07.719741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:07.782929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5bec5e-98cf-4065-b3d7-ae3f19bb4937 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:07.787978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:07.788591 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:07.788879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:07.789185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] PATH_INFO: `/v3/users/a243271ff1b54fa690c506beb080061c` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:07.790278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a243271ff1b54fa690c506beb080061c)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:07.792178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:07.836969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a243271ff1b54fa690c506beb080061c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:07.849074 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a243271ff1b54fa690c506beb080061c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:07.855782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a243271ff1b54fa690c506beb080061c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:07.856439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a243271ff1b54fa690c506beb080061c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:07.857071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a243271ff1b54fa690c506beb080061c'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:07.869578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1312/2617] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:07 2022] DELETE /identity/v3/users/a243271ff1b54fa690c506beb080061c => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:07.874243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:07.874532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:07.874809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:07.894855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:07.896049 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:07.960055 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1306/2618] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:07.969835 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:07.971240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:08.033103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4acf8e-e051-45ab-b630-26906920a99c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:08.036664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:08.037280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:08.037575 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:08.037829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:08.047481 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:08.048938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:08.058311 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1313/2619] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:08.378072 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:08.379499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:08.391539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e058a3a-53cb-495c-b135-63b33a94bb02 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:08.396192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:08.396931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:08.397282 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:08.397888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] PATH_INFO: `/v3/projects/ad2e16c79a244de9a8a73a55be7de22a` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:08.399500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad2e16c79a244de9a8a73a55be7de22a)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:08.401798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:08.448488 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad2e16c79a244de9a8a73a55be7de22a', 'request_id': 'req-f0ead00d-bca4-4836-8c25-f824749d64a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.456078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad2e16c79a244de9a8a73a55be7de22a', 'request_id': 'req-f0ead00d-bca4-4836-8c25-f824749d64a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.462415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad2e16c79a244de9a8a73a55be7de22a', 'request_id': 'req-f0ead00d-bca4-4836-8c25-f824749d64a2'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.472541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] The token cache is being invalidate because project ad2e16c79a244de9a8a73a55be7de22a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:08.472829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.473489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.474952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1307/2620] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:08 2022] DELETE /identity/v3/projects/ad2e16c79a244de9a8a73a55be7de22a => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:08.479008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:08.489159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504887b9-80a9-42a9-9126-f1f40ac8254f service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:08.507169 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:08.547060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:08.547626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:08.547911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:08.548272 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] PATH_INFO: `/v3/users/86b75ed88910450e9d78392c3332cf68` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:08.549847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86b75ed88910450e9d78392c3332cf68)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:08.551771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:08.605370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86b75ed88910450e9d78392c3332cf68'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.618526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86b75ed88910450e9d78392c3332cf68'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.625674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86b75ed88910450e9d78392c3332cf68'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.626434 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86b75ed88910450e9d78392c3332cf68'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.627036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-30bcbd23-8856-4f42-8cc6-23bb6917c0d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86b75ed88910450e9d78392c3332cf68'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:08.641495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1314/2621] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:08 2022] DELETE /identity/v3/users/86b75ed88910450e9d78392c3332cf68 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:08.920621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:08.922168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:08.935528 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ead00d-bca4-4836-8c25-f824749d64a2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:08.993847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:08.994483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:08.994760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:08.995032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] PATH_INFO: `/v3/projects/651575866eb04cab96ea93baeb65f0c4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:08.996168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=651575866eb04cab96ea93baeb65f0c4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:08.998077 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.028850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:09.046112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '651575866eb04cab96ea93baeb65f0c4', 'request_id': 'req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:09.053541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '651575866eb04cab96ea93baeb65f0c4', 'request_id': 'req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:09.059460 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '651575866eb04cab96ea93baeb65f0c4', 'request_id': 'req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:09.070409 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] The token cache is being invalidate because project 651575866eb04cab96ea93baeb65f0c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:09.070702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:09.071344 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f51983c-a58c-4ad7-8f85-5f2f56aa43c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:09.072888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1308/2622] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:08 2022] DELETE /identity/v3/projects/651575866eb04cab96ea93baeb65f0c4 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:09.077732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.078025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.078307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.098478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:09.099690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:09.160133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1315/2623] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:09.167568 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.167911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.168182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.187731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:09.189029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:09.197193 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1309/2624] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:09.201335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.211579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4398674e-3899-4aa6-97d1-0493d4cfa01f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.216125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.216746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.217059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.217334 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.218007 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:09.218310 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.219921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.231652 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1316/2625] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:09.235763 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.244966 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661d6a9f-0be7-4f61-b2db-0d9978631887 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.249701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.250323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.250600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.250871 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.251506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.253431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.280494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1310/2626] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/projects => generated 387 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:09.285041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.294789 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88735ff-65f6-418f-9e93-ac015fc6485d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.320219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:09.353519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.354474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.354924 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.355362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.356027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.357974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.381801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1317/2627] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/users => generated 294 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:09.383334 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.394172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66e2ca-4610-4578-91fe-79b24eb2f90a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.399045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.399757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.400109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.400394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.401123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.402713 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.414148 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1311/2628] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:09.418403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.428538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e899da7-8179-473b-ae9e-c2bc3e834f43 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.433119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddc70e2-c017-473f-8a66-a00cef193a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.433713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc70e2-c017-473f-8a66-a00cef193a6f admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.433990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc70e2-c017-473f-8a66-a00cef193a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.434258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc70e2-c017-473f-8a66-a00cef193a6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.435000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddc70e2-c017-473f-8a66-a00cef193a6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.436590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddc70e2-c017-473f-8a66-a00cef193a6f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.448031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1318/2629] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:09.452382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.462887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d0fadc-cdfd-46e1-a76a-b8f0594eb151 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.467583 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.468233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.468515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.468791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] PATH_INFO: `/v3/projects/13c6113bbc7e46459ee0d0eda4fbf7b0/users/890b8162f55944fa9d6a23c2a3ab797a/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.491981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] RBAC: Authorizing `identity:create_grant(project_id=13c6113bbc7e46459ee0d0eda4fbf7b0, user_id=890b8162f55944fa9d6a23c2a3ab797a, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.494991 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.509295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1312/2630] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:09 2022] PUT /identity/v3/projects/13c6113bbc7e46459ee0d0eda4fbf7b0/users/890b8162f55944fa9d6a23c2a3ab797a/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:09.520284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.520791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.520916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.547930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] MFA Rules not processed for user `890b8162f55944fa9d6a23c2a3ab797a`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:09.549178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:09.611881 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1319/2631] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:09.616327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.677994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70573204-7407-4d73-b441-65acaa0e493a admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.682681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.683286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.683569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.683845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.684486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.686428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.719430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.721719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1313/2632] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/projects => generated 387 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:09.730681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c5ffcc-d480-4d69-ab35-a4729c2bec79 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.798239 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.798854 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.799139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.799922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.800681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.802671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.827698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1320/2633] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:09 2022] POST /identity/v3/users => generated 300 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:09.831356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.842244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c35e8a2-207a-4a43-bfba-af9af7f1ca46 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.848081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.848817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.849536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.849536 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.850265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.851937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.866466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1314/2634] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:09.867999 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.879070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d23961c-b55d-44e5-9c96-88fb50a80c68 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.884170 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:09.885313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:09.885670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:09.886050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] PATH_INFO: `/v3/projects/830bdd93b94943c2ac51c522c24f326f/users/c9cabed741cc4d5aa21d208b06eed693/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:09.912075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=830bdd93b94943c2ac51c522c24f326f, user_id=c9cabed741cc4d5aa21d208b06eed693, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:09.914940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:09.928440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1321/2635] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:09 2022] PUT /identity/v3/projects/830bdd93b94943c2ac51c522c24f326f/users/c9cabed741cc4d5aa21d208b06eed693/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:09.932382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:09.943472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a3d7c-ad84-4ee1-993e-f270fca734a4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:09.964280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:10.003175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794bd8e9-e6d4-4d46-817d-7cfec10a6c53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:10.003782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794bd8e9-e6d4-4d46-817d-7cfec10a6c53 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.004063 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794bd8e9-e6d4-4d46-817d-7cfec10a6c53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.004336 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794bd8e9-e6d4-4d46-817d-7cfec10a6c53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.005010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794bd8e9-e6d4-4d46-817d-7cfec10a6c53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:10.006618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794bd8e9-e6d4-4d46-817d-7cfec10a6c53 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:10.016524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1315/2636] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:09 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:10.021381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:10.026823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:10.033487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d9807-6954-4d90-bdb2-5ff688487e42 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:10.039381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:10.040146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.040429 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.040705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] PATH_INFO: `/v3/projects/830bdd93b94943c2ac51c522c24f326f/users/c9cabed741cc4d5aa21d208b06eed693/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.042685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] RBAC: Authorizing `identity:create_grant(project_id=830bdd93b94943c2ac51c522c24f326f, user_id=c9cabed741cc4d5aa21d208b06eed693, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:10.045393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:10.055923 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1322/2637] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:10 2022] PUT /identity/v3/projects/830bdd93b94943c2ac51c522c24f326f/users/c9cabed741cc4d5aa21d208b06eed693/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:10.067512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.067801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.068074 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.095807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] MFA Rules not processed for user `c9cabed741cc4d5aa21d208b06eed693`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:10.097547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:10.167995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1316/2638] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:10 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:10.178694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:10.180457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:10.243243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0d7a3f-ad1c-4859-8999-f62b5a75427f admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:10.248590 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618bd178-c253-42d3-bc32-ac77664e644e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:10.249792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618bd178-c253-42d3-bc32-ac77664e644e service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.250070 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618bd178-c253-42d3-bc32-ac77664e644e service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.250417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618bd178-c253-42d3-bc32-ac77664e644e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.260491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618bd178-c253-42d3-bc32-ac77664e644e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:10.261964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618bd178-c253-42d3-bc32-ac77664e644e service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:10.272034 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1323/2639] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:10 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:10.422301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:10.484108 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee70326-6aa7-4f2d-9d1c-9a4a688166b3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:10.489361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:10.489992 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.490272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.490545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] PATH_INFO: `/v3/users/890b8162f55944fa9d6a23c2a3ab797a` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.491812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] RBAC: Authorizing `identity:delete_user(user_id=890b8162f55944fa9d6a23c2a3ab797a)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:10.493794 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:10.559331 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '890b8162f55944fa9d6a23c2a3ab797a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:10.561495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:10.573290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '890b8162f55944fa9d6a23c2a3ab797a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:10.574915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '890b8162f55944fa9d6a23c2a3ab797a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:10.575691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '890b8162f55944fa9d6a23c2a3ab797a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:10.613520 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '890b8162f55944fa9d6a23c2a3ab797a'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:10.648138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1317/2640] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:10 2022] DELETE /identity/v3/users/890b8162f55944fa9d6a23c2a3ab797a => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:10.650084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.650951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.651269 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.675714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:10.676970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:10.756245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1324/2641] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:10.765952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:10.767464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:10.857251 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583f1f4-4735-4ceb-95c1-19d4054e0938 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:10.861740 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:10.862711 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:10.862998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:10.863272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:10.873281 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:10.874731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:10.884681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1318/2642] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:11.264393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:11.266609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:11.280463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768f5ffa-7f1f-447d-9f1c-856947fcee93 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:11.285722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:11.286431 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:11.286776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:11.287343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] PATH_INFO: `/v3/projects/13c6113bbc7e46459ee0d0eda4fbf7b0` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:11.289426 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=13c6113bbc7e46459ee0d0eda4fbf7b0)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:11.291623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:11.345543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13c6113bbc7e46459ee0d0eda4fbf7b0', 'request_id': 'req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.353322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13c6113bbc7e46459ee0d0eda4fbf7b0', 'request_id': 'req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.363285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13c6113bbc7e46459ee0d0eda4fbf7b0', 'request_id': 'req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.375037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] The token cache is being invalidate because project 13c6113bbc7e46459ee0d0eda4fbf7b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:11.375327 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.375990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.378189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1325/2643] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:11 2022] DELETE /identity/v3/projects/13c6113bbc7e46459ee0d0eda4fbf7b0 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:11.382297 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:11.392190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3af6593-7654-4ace-98b2-d3db1446bb44 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:11.456651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:11.457390 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:11.457600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:11.457875 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] PATH_INFO: `/v3/users/c9cabed741cc4d5aa21d208b06eed693` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:11.459024 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9cabed741cc4d5aa21d208b06eed693)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:11.461011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:11.514423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9cabed741cc4d5aa21d208b06eed693'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.521714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9cabed741cc4d5aa21d208b06eed693'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.522434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9cabed741cc4d5aa21d208b06eed693'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.523039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9cabed741cc4d5aa21d208b06eed693'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.533085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-d1f924f4-1bda-4c2d-8c08-fcb10932dd5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9cabed741cc4d5aa21d208b06eed693'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.538886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:11.553269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1319/2644] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:11 2022] DELETE /identity/v3/users/c9cabed741cc4d5aa21d208b06eed693 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:11.851380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:11.860491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:11.873705 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6dd0dc-6abc-46f2-b3e1-8b51610812d4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:11.886200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:11.935357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:11.935978 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:11.936254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:11.936526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] PATH_INFO: `/v3/projects/830bdd93b94943c2ac51c522c24f326f` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:11.937670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=830bdd93b94943c2ac51c522c24f326f)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:11.940491 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:11.987583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '830bdd93b94943c2ac51c522c24f326f', 'request_id': 'req-2084d688-89cf-4daa-aaa0-7f3de3d240d7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:11.995945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '830bdd93b94943c2ac51c522c24f326f', 'request_id': 'req-2084d688-89cf-4daa-aaa0-7f3de3d240d7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:12.001770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '830bdd93b94943c2ac51c522c24f326f', 'request_id': 'req-2084d688-89cf-4daa-aaa0-7f3de3d240d7'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:12.014829 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] The token cache is being invalidate because project 830bdd93b94943c2ac51c522c24f326f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:12.015136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:12.015787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2084d688-89cf-4daa-aaa0-7f3de3d240d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:12.017342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1326/2645] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:11 2022] DELETE /identity/v3/projects/830bdd93b94943c2ac51c522c24f326f => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:12.022737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.023132 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.023412 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.043005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:12.044552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:12.102793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1320/2646] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:12.110414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.110700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.110974 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.132144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:12.133333 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:12.141770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1327/2647] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:12.147189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.157092 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558174e-6028-4539-abfd-36ceb9df24f9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.164267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.164267 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.164490 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.164725 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.165475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:12.165591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.167660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.180953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1321/2648] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:12.185362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.196221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51a5a77-80c1-4749-a089-0effd3d1b5df None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.201444 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.202049 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.202341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.202576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.203195 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.205104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.235449 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1328/2649] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:12 2022] POST /identity/v3/projects => generated 371 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:18:12.240127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.250545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533c9cd7-2184-4f71-8898-ffdb605524b1 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.316027 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.316632 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.316939 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.317211 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.317830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.319746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.342186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1322/2650] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:12 2022] POST /identity/v3/users => generated 286 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:12.347102 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.357696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d5bc47-8f02-48b0-b77c-63ade365b3f2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.363075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.363676 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.363955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.364227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.364884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.366490 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.382894 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1329/2651] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:12.384547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.395990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d079307-fafc-460b-9f24-0c7b20511a84 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.402439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af2d9ac-ca78-4ddd-b82d-b60e178367ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.403162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af2d9ac-ca78-4ddd-b82d-b60e178367ab admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.403439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af2d9ac-ca78-4ddd-b82d-b60e178367ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.403708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af2d9ac-ca78-4ddd-b82d-b60e178367ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.404350 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af2d9ac-ca78-4ddd-b82d-b60e178367ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.406171 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af2d9ac-ca78-4ddd-b82d-b60e178367ab admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.546205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1323/2652] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:12 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:12.551181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.561122 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e1212-333e-46ee-8b9b-e61e8aff15d8 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.565944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.566582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.566875 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.567129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] PATH_INFO: `/v3/projects/763e7697890b447c845df94bde88d83d/users/46a87800acb5474f9acc4438b33cd492/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.582112 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=763e7697890b447c845df94bde88d83d, user_id=46a87800acb5474f9acc4438b33cd492, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.584888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.598879 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1330/2653] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:12 2022] PUT /identity/v3/projects/763e7697890b447c845df94bde88d83d/users/46a87800acb5474f9acc4438b33cd492/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:12.604482 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f2c2d8-2028-4085-821e-e117bc935364 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.604728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f2c2d8-2028-4085-821e-e117bc935364 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.605059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f2c2d8-2028-4085-821e-e117bc935364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.625010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-83f2c2d8-2028-4085-821e-e117bc935364 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:12.626329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-83f2c2d8-2028-4085-821e-e117bc935364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:12.673042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:12.688484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1324/2654] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:12.698352 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:12.700091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:12.746338 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:12.759390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73807f38-ca44-4043-9b1d-c8e673a529fd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:12.763083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:12.763685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:12.763968 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:12.764250 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:12.774656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:12.776124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:12.786420 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1331/2655] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:13.407372 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.407663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.407940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.443641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] MFA Rules not processed for user `46a87800acb5474f9acc4438b33cd492`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:13.444898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:13.526736 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1325/2656] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:13 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:13.527822 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:13.539941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5172ef46-9e8c-4b22-a125-eb0e0adf261b service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:13.546855 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:13.547465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.547775 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.548052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.548698 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:13.550743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:13.618691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:13.626823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1332/2657] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:13 2022] POST /identity/v3/projects => generated 373 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:18:13.632506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0600412d-ed7b-49b3-a36b-50fad8a399b1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:13.695625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:13.696240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.696519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.696790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.697437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:13.699309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:13.720381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1326/2658] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:13 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:13.724372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:13.734467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c51740-79ca-4ef3-9eed-b541bf1363c3 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:13.740180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:13.740820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.741130 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.741407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.742147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:13.743886 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:13.755995 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1333/2659] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:13.761385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:13.771552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ef13d-7a3f-4c0c-861c-0269683f0c22 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:13.776190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:13.776825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.777137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.777408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] PATH_INFO: `/v3/projects/1d43731206824351bac01062fa1d0f81/users/a5de3972eef34c938b551fe3cdf1d30b/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.791599 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d43731206824351bac01062fa1d0f81, user_id=a5de3972eef34c938b551fe3cdf1d30b, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:13.794230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:13.807550 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1327/2660] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:13 2022] PUT /identity/v3/projects/1d43731206824351bac01062fa1d0f81/users/a5de3972eef34c938b551fe3cdf1d30b/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:13.811814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:13.823842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51db551-c25e-4873-91e3-8db7acd19a80 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:13.882868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91e18b-6ebd-4499-bb80-fe99114fd17d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:13.883476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91e18b-6ebd-4499-bb80-fe99114fd17d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.883760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91e18b-6ebd-4499-bb80-fe99114fd17d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.884290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91e18b-6ebd-4499-bb80-fe99114fd17d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.885052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91e18b-6ebd-4499-bb80-fe99114fd17d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:13.886790 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91e18b-6ebd-4499-bb80-fe99114fd17d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:13.896768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1334/2661] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:13 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:13.901230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:13.913137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad1c771-af6d-4dc6-b220-21a0a1e6cc51 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:13.917775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:13.918444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:13.918720 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:13.918996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] PATH_INFO: `/v3/projects/1d43731206824351bac01062fa1d0f81/users/a5de3972eef34c938b551fe3cdf1d30b/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:13.921026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d43731206824351bac01062fa1d0f81, user_id=a5de3972eef34c938b551fe3cdf1d30b, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:13.924088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:13.936514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1328/2662] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:13 2022] PUT /identity/v3/projects/1d43731206824351bac01062fa1d0f81/users/a5de3972eef34c938b551fe3cdf1d30b/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:14.538503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:14.539058 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:14.539167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:14.568076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] MFA Rules not processed for user `a5de3972eef34c938b551fe3cdf1d30b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:14.569259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:14.652955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1335/2663] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:14 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:14.663203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:14.664977 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:14.709665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:14.733603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bb6914-664d-427d-88ce-4c6ff874b6e3 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:14.737760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:14.738370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:14.738649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:14.738923 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:14.805241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:14.806690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:14.826405 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1329/2664] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:14 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:16.163463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:16.198763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:16.205286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:16.214245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729463d7-0b84-4e6b-8977-be31173974ae service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:16.217827 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:16.218415 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:16.218690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:16.218957 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:16.222155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48cc65-1ee6-4197-8851-dfbebad8639d None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:16.225771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:16.226374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:16.226653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:16.226926 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:16.228066 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:16.228670 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:16.229524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:16.239471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1330/2665] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:16 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:18:16.290801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:16.292259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:16.302372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1336/2666] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:19.698543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:19.699981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:19.714316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b6cccc-7ee9-4f66-aa12-7114884142bb service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:19.721730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:19.722834 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:19.723228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:19.723379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] PATH_INFO: `/v3/users/f16c1cde815c4a2580cd5ebf7b3ed76d` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:19.724698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f16c1cde815c4a2580cd5ebf7b3ed76d)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:19.726829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:19.777858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f16c1cde815c4a2580cd5ebf7b3ed76d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:19.785309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f16c1cde815c4a2580cd5ebf7b3ed76d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:19.786075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f16c1cde815c4a2580cd5ebf7b3ed76d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:19.786769 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f16c1cde815c4a2580cd5ebf7b3ed76d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:19.794205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e307c278-39e2-4dd5-b027-c331409d6938 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f16c1cde815c4a2580cd5ebf7b3ed76d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:19.818158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1331/2667] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:19 2022] DELETE /identity/v3/users/f16c1cde815c4a2580cd5ebf7b3ed76d => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:19.988057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:19.998134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65c7e23-2be6-4a38-9e4d-ea1450b55e48 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.055863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.056485 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.056761 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.057060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] PATH_INFO: `/v3/projects/cc20488e0b2d48f796052c4e76990b19` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.058158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc20488e0b2d48f796052c4e76990b19)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.060040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.104993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc20488e0b2d48f796052c4e76990b19', 'request_id': 'req-6037944b-322e-4711-8f4f-0ce7ccc59451'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:20.112520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc20488e0b2d48f796052c4e76990b19', 'request_id': 'req-6037944b-322e-4711-8f4f-0ce7ccc59451'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:20.117885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc20488e0b2d48f796052c4e76990b19', 'request_id': 'req-6037944b-322e-4711-8f4f-0ce7ccc59451'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:20.129076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] The token cache is being invalidate because project cc20488e0b2d48f796052c4e76990b19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:20.129360 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:20.129980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6037944b-322e-4711-8f4f-0ce7ccc59451 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:20.131428 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1337/2668] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:19 2022] DELETE /identity/v3/projects/cc20488e0b2d48f796052c4e76990b19 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:20.137648 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.138016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.138314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.157981 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:20.159235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:20.207352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:20.225668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1332/2669] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:20.233399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.233888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.234237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.254550 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:20.255911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:20.265685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1338/2670] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:20.270505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.280529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb198f-db97-438f-8819-a89abb6a6b46 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.285190 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.285792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.286070 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.286342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.287073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:20.287313 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.289220 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.302184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1333/2671] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:20.307080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.317319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6ec5c3-6f53-43a6-ab90-240d10eeea17 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.322910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.323506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.323801 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.324050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.324678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.326600 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.332119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:20.357929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1339/2672] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:20.363333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.373863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f2107-58b9-4f6d-a360-2cc2c81042d2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.445338 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.445987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.446269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.446545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.447176 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.449492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.473728 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1334/2673] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/users => generated 291 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:20.477784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.488623 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58ced49-8f93-4714-beaa-9940cf8ac50d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.493527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.494132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.494406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.494672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.495336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.496944 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.508247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1340/2674] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:20.512145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.522620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786a49a9-8117-4984-ab25-ebc32dbba522 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.527724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10e8e91-7fb4-421c-bf52-4c697f3a7b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.528316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10e8e91-7fb4-421c-bf52-4c697f3a7b7e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.528593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10e8e91-7fb4-421c-bf52-4c697f3a7b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.528929 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10e8e91-7fb4-421c-bf52-4c697f3a7b7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.529651 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10e8e91-7fb4-421c-bf52-4c697f3a7b7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.531393 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10e8e91-7fb4-421c-bf52-4c697f3a7b7e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.542255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1335/2675] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:20.546355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.575467 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09dcc46f-2499-4218-84a9-7f6021f6f55b admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.580987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.581653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.581927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.582208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] PATH_INFO: `/v3/projects/117864df2d3c43339dfa6aacc657672d/users/09dbd3184120429e8e5ce92db986177e/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.602414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=117864df2d3c43339dfa6aacc657672d, user_id=09dbd3184120429e8e5ce92db986177e, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.605060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.617381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1341/2676] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:20 2022] PUT /identity/v3/projects/117864df2d3c43339dfa6aacc657672d/users/09dbd3184120429e8e5ce92db986177e/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:20.627590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.627880 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.628179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.655308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] MFA Rules not processed for user `09dbd3184120429e8e5ce92db986177e`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:20.656495 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:20.718686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1336/2677] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:20.723336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.783842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e75cd4-94a5-4b78-b51f-3fbb4f291c7d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.788364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.788989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.789268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.789564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.790180 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.792066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.818273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1342/2678] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/projects => generated 381 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:20.822577 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.833776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b67d9-3f2f-416e-ba8c-ed90cef317f4 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.878098 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:20.895258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.896026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.896356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.896633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.897326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.899239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.925509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1337/2679] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:20 2022] POST /identity/v3/users => generated 297 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:18:20.929462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.941394 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011a28-5450-493a-ae3c-7a3e7df6da70 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.946065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.946741 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.947083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.947439 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.948145 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.949843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.960914 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1343/2680] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:20.964931 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.975682 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8803ce10-eb74-4c3f-8675-f0034f88adfe admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.980380 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.981191 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.981532 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.981865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] PATH_INFO: `/v3/projects/e787a418532c4b34a410e1757fdc28d1/users/0c1c2bedfe984048bb395d1af356628d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.997064 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e787a418532c4b34a410e1757fdc28d1, user_id=0c1c2bedfe984048bb395d1af356628d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.999742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:21.012884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1338/2681] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:20 2022] PUT /identity/v3/projects/e787a418532c4b34a410e1757fdc28d1/users/0c1c2bedfe984048bb395d1af356628d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:21.016544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:21.027090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00c9e30-76d3-4975-be01-c7fe0b4b96ce admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:21.083636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56017f7-9001-48b4-a621-81cd0ab8c896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:21.084236 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56017f7-9001-48b4-a621-81cd0ab8c896 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.084519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56017f7-9001-48b4-a621-81cd0ab8c896 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.084786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56017f7-9001-48b4-a621-81cd0ab8c896 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.085450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56017f7-9001-48b4-a621-81cd0ab8c896 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:21.087022 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56017f7-9001-48b4-a621-81cd0ab8c896 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:21.097458 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1344/2682] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:21 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:21.101612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:21.111570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24851fb-a43c-4e9a-a27a-43cae536bec4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:21.116142 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:21.116776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.117082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.117356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] PATH_INFO: `/v3/projects/e787a418532c4b34a410e1757fdc28d1/users/0c1c2bedfe984048bb395d1af356628d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.119269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e787a418532c4b34a410e1757fdc28d1, user_id=0c1c2bedfe984048bb395d1af356628d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:21.121888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:21.131755 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1339/2683] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:21 2022] PUT /identity/v3/projects/e787a418532c4b34a410e1757fdc28d1/users/0c1c2bedfe984048bb395d1af356628d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:21.143537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.143823 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.144100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.172480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] MFA Rules not processed for user `0c1c2bedfe984048bb395d1af356628d`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:21.173728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:21.181699 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:21.241988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1345/2684] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:21 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:21.253836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:21.255598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:21.312173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eef911-2859-4474-878d-b2125a46c81c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:21.316113 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:21.316701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.317011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.317283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.325999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:21.327452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:21.337166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1340/2685] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:21 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:21.410401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:21.420496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26783239-e8d3-4b97-b388-e632546454f0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:21.424215 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7345bcfd-5fc5-4790-ac20-0dd74bc4339c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:21.424804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345bcfd-5fc5-4790-ac20-0dd74bc4339c service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.425113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345bcfd-5fc5-4790-ac20-0dd74bc4339c service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.425496 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7345bcfd-5fc5-4790-ac20-0dd74bc4339c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.482681 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7345bcfd-5fc5-4790-ac20-0dd74bc4339c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:21.484124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7345bcfd-5fc5-4790-ac20-0dd74bc4339c service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:21.493354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1346/2686] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:21 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:21.770068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:21.831017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad0fca-048f-4390-8ead-365f9c651097 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:21.835641 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:21.836265 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.836545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.836816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] PATH_INFO: `/v3/users/09dbd3184120429e8e5ce92db986177e` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.837961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=09dbd3184120429e8e5ce92db986177e)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:21.839867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:21.881969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:21.887702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09dbd3184120429e8e5ce92db986177e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:21.895998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09dbd3184120429e8e5ce92db986177e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:21.896663 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09dbd3184120429e8e5ce92db986177e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:21.897369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09dbd3184120429e8e5ce92db986177e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:21.904614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09dbd3184120429e8e5ce92db986177e'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:21.923441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1341/2687] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:21 2022] DELETE /identity/v3/users/09dbd3184120429e8e5ce92db986177e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:21.928062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:21.928354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:21.928643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:21.947946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:21.949212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:22.008836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1347/2688] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:22.018383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:22.019830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:22.078233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e77ded0-0c61-4a3f-a446-587ae26d3c9e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:22.081969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:22.082654 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:22.082940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:22.083214 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:22.092436 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:22.093886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:22.103982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1342/2689] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:22.634692 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:22.636174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:22.651167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06ab4a3-5cb0-40e2-b932-de7d12e1d0fa None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:22.655907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:22.656581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:22.656952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:22.657237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] PATH_INFO: `/v3/projects/117864df2d3c43339dfa6aacc657672d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:22.659062 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] RBAC: Authorizing `identity:delete_project(project_id=117864df2d3c43339dfa6aacc657672d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:22.661097 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:22.706474 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:22.714684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '117864df2d3c43339dfa6aacc657672d', 'request_id': 'req-805ac6e4-2059-40a3-b6b6-d0975b49deff'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.722964 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '117864df2d3c43339dfa6aacc657672d', 'request_id': 'req-805ac6e4-2059-40a3-b6b6-d0975b49deff'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.728512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '117864df2d3c43339dfa6aacc657672d', 'request_id': 'req-805ac6e4-2059-40a3-b6b6-d0975b49deff'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.739303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] The token cache is being invalidate because project 117864df2d3c43339dfa6aacc657672d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:22.739583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.740225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.741732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1348/2690] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:22 2022] DELETE /identity/v3/projects/117864df2d3c43339dfa6aacc657672d => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:22.745730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:22.755053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e1a255-73e7-48a5-b627-2cc221a38904 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:22.813717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:22.814603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:22.814857 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:22.815121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] PATH_INFO: `/v3/users/0c1c2bedfe984048bb395d1af356628d` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:22.816248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c1c2bedfe984048bb395d1af356628d)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:22.818236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:22.866661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c1c2bedfe984048bb395d1af356628d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.874636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c1c2bedfe984048bb395d1af356628d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.875311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c1c2bedfe984048bb395d1af356628d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.875935 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c1c2bedfe984048bb395d1af356628d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.882866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6e15fca0-8e78-4941-a753-9de6dec1ecf5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c1c2bedfe984048bb395d1af356628d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:22.884312 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:22.904501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1343/2691] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:22 2022] DELETE /identity/v3/users/0c1c2bedfe984048bb395d1af356628d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:23.209589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.211205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:23.227268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ac6e4-2059-40a3-b6b6-d0975b49deff admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.287292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.288078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.288361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.288633 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] PATH_INFO: `/v3/projects/e787a418532c4b34a410e1757fdc28d1` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.289951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e787a418532c4b34a410e1757fdc28d1)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.292147 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.342736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e787a418532c4b34a410e1757fdc28d1', 'request_id': 'req-2b7a4b59-b8c2-4990-97af-9985246fff24'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:23.351088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e787a418532c4b34a410e1757fdc28d1', 'request_id': 'req-2b7a4b59-b8c2-4990-97af-9985246fff24'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:23.356811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e787a418532c4b34a410e1757fdc28d1', 'request_id': 'req-2b7a4b59-b8c2-4990-97af-9985246fff24'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:23.368418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] The token cache is being invalidate because project e787a418532c4b34a410e1757fdc28d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:23.368628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:23.369283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-2b7a4b59-b8c2-4990-97af-9985246fff24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:23.370776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1349/2692] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:23 2022] DELETE /identity/v3/projects/e787a418532c4b34a410e1757fdc28d1 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:23.377171 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.377457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.377729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.397789 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:23.399014 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:23.458354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1344/2693] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:23.466213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.466500 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.466776 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.485840 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:23.487709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:23.495520 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1350/2694] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:23.499627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.509951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87774dad-46a6-42fd-9ffd-89855c2aeb9b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.514503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.515093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.515371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.515636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.516271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:23.516572 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.518716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.529810 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1345/2695] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:23.534535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.543630 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99129b77-54ff-4757-9313-a2d2f5195347 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.548212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.548805 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.549273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.549543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.550172 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.554878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.587820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1351/2696] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:23 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:23.591791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.602462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23095a3c-1373-4bc2-8853-508678661eda admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.657503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.658020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.658879 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.659150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.659775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.661712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.682382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1346/2697] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:23 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:23.686433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.697367 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4320bdbf-8558-42df-832f-aabc6e41909c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.701867 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.702452 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.702727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.702993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.703627 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.705859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.715842 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1352/2698] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:23.719542 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.729363 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a8b2de-a68c-4546-8512-334440ff04bd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.734057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e718b3e-e437-4fce-a941-2263c5c4503a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.734671 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e718b3e-e437-4fce-a941-2263c5c4503a admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.734952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e718b3e-e437-4fce-a941-2263c5c4503a admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.735377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e718b3e-e437-4fce-a941-2263c5c4503a admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.735859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e718b3e-e437-4fce-a941-2263c5c4503a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.737515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e718b3e-e437-4fce-a941-2263c5c4503a admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.751832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1347/2699] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:23.756493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.766869 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac5ec98-7ce2-42ed-89cb-75505ba4f196 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.772063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.772709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.773101 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.773382 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] PATH_INFO: `/v3/projects/fa2995c95af34d6b97ec0c61915d9edc/users/274cc054c42143bc8d55256e4372bea6/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.787581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa2995c95af34d6b97ec0c61915d9edc, user_id=274cc054c42143bc8d55256e4372bea6, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.790287 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:23.792465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:23.804220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1353/2700] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:23 2022] PUT /identity/v3/projects/fa2995c95af34d6b97ec0c61915d9edc/users/274cc054c42143bc8d55256e4372bea6/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:23.816150 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.816625 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.816746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.843995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] MFA Rules not processed for user `274cc054c42143bc8d55256e4372bea6`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:23.845255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:23.878173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:23.910685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1348/2701] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:23.922231 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:23.924626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:23.980222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8f6b6d-f6d6-4fb0-8c6a-77093020aa88 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:23.983888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df800d87-7b54-4083-89fb-8f7db08ec59b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:23.984477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df800d87-7b54-4083-89fb-8f7db08ec59b service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:23.984753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df800d87-7b54-4083-89fb-8f7db08ec59b service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:23.985053 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df800d87-7b54-4083-89fb-8f7db08ec59b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:23.993720 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df800d87-7b54-4083-89fb-8f7db08ec59b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:23.995134 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df800d87-7b54-4083-89fb-8f7db08ec59b service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:24.004205 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1354/2702] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:26.569519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:26.632262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a148833-5cf9-42e5-ad2d-a9bfc742f15f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:26.636848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:26.637501 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:26.637779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:26.638050 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] PATH_INFO: `/v3/users/274cc054c42143bc8d55256e4372bea6` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:26.639168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] RBAC: Authorizing `identity:delete_user(user_id=274cc054c42143bc8d55256e4372bea6)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:26.641104 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:26.687192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '274cc054c42143bc8d55256e4372bea6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:26.695016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '274cc054c42143bc8d55256e4372bea6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:26.695689 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '274cc054c42143bc8d55256e4372bea6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:26.696307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '274cc054c42143bc8d55256e4372bea6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:26.703751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '274cc054c42143bc8d55256e4372bea6'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:26.721751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1349/2703] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:26 2022] DELETE /identity/v3/users/274cc054c42143bc8d55256e4372bea6 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:26.726266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:26.726723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:26.726841 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:26.750343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:26.751785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:26.813017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1355/2704] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:26.828896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:26.831701 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:26.913383 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f2d8d3-b950-4ed4-a514-eb5ef764f199 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:26.918537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd5ef0-d120-451d-bcf5-bd358aeda20c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:26.919376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd5ef0-d120-451d-bcf5-bd358aeda20c service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:26.919795 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd5ef0-d120-451d-bcf5-bd358aeda20c service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:26.920121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd5ef0-d120-451d-bcf5-bd358aeda20c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:26.931151 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd5ef0-d120-451d-bcf5-bd358aeda20c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:26.932685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd5ef0-d120-451d-bcf5-bd358aeda20c service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:26.937388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:26.948884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1350/2705] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:27.250446 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.253225 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:27.267183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d14d5a-9fd0-4095-9215-9647026f2c12 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.273248 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.274160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.274658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.275157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] PATH_INFO: `/v3/projects/fa2995c95af34d6b97ec0c61915d9edc` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.276514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa2995c95af34d6b97ec0c61915d9edc)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.279051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.334882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa2995c95af34d6b97ec0c61915d9edc', 'request_id': 'req-f76f8278-ac5a-4b30-917a-86f784387460'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:27.343774 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa2995c95af34d6b97ec0c61915d9edc', 'request_id': 'req-f76f8278-ac5a-4b30-917a-86f784387460'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:27.351316 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa2995c95af34d6b97ec0c61915d9edc', 'request_id': 'req-f76f8278-ac5a-4b30-917a-86f784387460'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:27.363149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] The token cache is being invalidate because project fa2995c95af34d6b97ec0c61915d9edc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:27.363442 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:27.364099 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f76f8278-ac5a-4b30-917a-86f784387460 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:27.365618 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1356/2706] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:27 2022] DELETE /identity/v3/projects/fa2995c95af34d6b97ec0c61915d9edc => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:27.370354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.370798 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.371137 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.390836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:27.392039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:27.451432 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1351/2707] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:27.459026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.459309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.459583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.479852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:27.481127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:27.490719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1357/2708] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:27.495005 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.504616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0f13c-67d3-45f7-9ca8-eda9ac1fccbc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.509275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.509855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.510136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.510381 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.511022 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:27.511335 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.512903 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.524280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1352/2709] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:27.529643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.543666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9dbe7e-f0a0-4c91-babb-1df915832578 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.548292 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.550519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.550798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.551071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.553010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.555028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.570320 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:27.588150 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1358/2710] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:27 2022] POST /identity/v3/projects => generated 373 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:27.592192 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.602748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a490942-b6a1-410f-a7f3-01e1cce043f7 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.659876 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.660544 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.660821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.661122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.661747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.663850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.686508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1353/2711] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:27 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:27.691628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.702447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a35e7a9-a615-470d-bf4b-a1efef993672 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.708368 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.709003 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.709283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.709635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.710179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.711765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.724770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1359/2712] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:27.726748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.738242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72a1ce7-f94d-4124-91a5-bea3969b65ed admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.742955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30b7bc6-1eeb-4442-bddd-9351c42cb987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.743543 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30b7bc6-1eeb-4442-bddd-9351c42cb987 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.743819 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30b7bc6-1eeb-4442-bddd-9351c42cb987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.744101 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30b7bc6-1eeb-4442-bddd-9351c42cb987 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.744714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30b7bc6-1eeb-4442-bddd-9351c42cb987 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.746307 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30b7bc6-1eeb-4442-bddd-9351c42cb987 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.757248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1354/2713] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:27.761565 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.771959 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735cef4a-013f-4fd4-a614-c9db440b9109 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:27.776991 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:27.777678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.777954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.778220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] PATH_INFO: `/v3/projects/3b547412b0c74316a1ab673830d48ab3/users/1a16f9f124444f43a689e269059cc523/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.792806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b547412b0c74316a1ab673830d48ab3, user_id=1a16f9f124444f43a689e269059cc523, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:27.795518 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:27.808937 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1360/2714] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:27 2022] PUT /identity/v3/projects/3b547412b0c74316a1ab673830d48ab3/users/1a16f9f124444f43a689e269059cc523/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:27.819800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e1181-a84d-483a-a5fa-41d109487300 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:27.820083 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e1181-a84d-483a-a5fa-41d109487300 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:27.820403 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476e1181-a84d-483a-a5fa-41d109487300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:27.848275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-476e1181-a84d-483a-a5fa-41d109487300 None None] MFA Rules not processed for user `1a16f9f124444f43a689e269059cc523`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:27.849717 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-476e1181-a84d-483a-a5fa-41d109487300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:27.914746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:27.930611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1355/2715] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:27.944465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:27.946384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:28.004973 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:28.005280 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:28.005533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:28.021174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b09aba-7c3b-4f48-8ff5-052afe012a24 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:28.026422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:28.027185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:28.027593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:28.027966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:28.028616 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:28.030323 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:28.039324 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:28.041227 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:28.053136 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1361/2716] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:27 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:28.083672 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:28.113646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1356/2717] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:18:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:28.131832 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:28.133811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:28.178766 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248d121-b946-49d5-ae0e-ba026018d891 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:28.190235 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:28.191477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:28.191981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:28.192480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] PATH_INFO: `/v3/users/1a16f9f124444f43a689e269059cc523` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:28.194664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a16f9f124444f43a689e269059cc523)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:28.197043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:28.245086 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef74d6a-437e-41a9-b073-4eb99cf90e69 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:28.251196 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079d6ccb-05aa-41e8-bc39-375109b625ef service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:28.252332 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d6ccb-05aa-41e8-bc39-375109b625ef service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:28.252610 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d6ccb-05aa-41e8-bc39-375109b625ef service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:28.252900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d6ccb-05aa-41e8-bc39-375109b625ef service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:28.263657 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079d6ccb-05aa-41e8-bc39-375109b625ef service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:28.265445 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079d6ccb-05aa-41e8-bc39-375109b625ef service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:28.281483 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1357/2718] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:18:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:18:28.288211 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a16f9f124444f43a689e269059cc523'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:28.300521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:28.303673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a16f9f124444f43a689e269059cc523'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:28.361435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a16f9f124444f43a689e269059cc523'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:28.362240 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a16f9f124444f43a689e269059cc523'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:28.362973 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a16f9f124444f43a689e269059cc523'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:28.388613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1362/2719] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:28 2022] DELETE /identity/v3/users/1a16f9f124444f43a689e269059cc523 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:28.393997 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:28.394664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:28.395041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:28.416806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:28.418400 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:28.516560 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1358/2720] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:28.528273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:28.529752 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:28.667722 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e859e2-9f26-4477-aa50-65422b3f0bf7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:28.673710 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:28.674563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:28.674940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:28.675308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:28.687687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:28.689616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:28.708665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1363/2721] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:29.264967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:29.268121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:29.295749 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e682df6-c00d-4f0f-85e9-318e49547b5f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:29.309474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:29.310130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:29.310408 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:29.310676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] PATH_INFO: `/v3/projects/3b547412b0c74316a1ab673830d48ab3` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:29.311811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b547412b0c74316a1ab673830d48ab3)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:29.313743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:29.345304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:29.375546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b547412b0c74316a1ab673830d48ab3', 'request_id': 'req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:29.391303 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b547412b0c74316a1ab673830d48ab3', 'request_id': 'req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:29.400458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b547412b0c74316a1ab673830d48ab3', 'request_id': 'req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:29.433687 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] The token cache is being invalidate because project 3b547412b0c74316a1ab673830d48ab3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:29.434119 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:29.435140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:29.477613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1359/2722] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:29 2022] DELETE /identity/v3/projects/3b547412b0c74316a1ab673830d48ab3 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:30.196665 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:30.198218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:30.225523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c721c612-f6f6-439b-90d3-2bead55564c9 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:30.302052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:30.303025 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:30.303728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:30.304212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] PATH_INFO: `/v3/users/46a87800acb5474f9acc4438b33cd492` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:30.305541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] RBAC: Authorizing `identity:delete_user(user_id=46a87800acb5474f9acc4438b33cd492)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:30.307546 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:30.371472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46a87800acb5474f9acc4438b33cd492'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.394636 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46a87800acb5474f9acc4438b33cd492'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.412093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46a87800acb5474f9acc4438b33cd492'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.413220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46a87800acb5474f9acc4438b33cd492'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.414084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46a87800acb5474f9acc4438b33cd492'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.428659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:30.453084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1364/2723] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:30 2022] DELETE /identity/v3/users/46a87800acb5474f9acc4438b33cd492 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:30.658607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:30.660444 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:30.678018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3cc22a-ef22-4386-b68b-5dd1f7bec481 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:30.795271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:30.795905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:30.796182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:30.796591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] PATH_INFO: `/v3/projects/763e7697890b447c845df94bde88d83d` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:30.797772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] RBAC: Authorizing `identity:delete_project(project_id=763e7697890b447c845df94bde88d83d)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:30.800224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:30.894273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '763e7697890b447c845df94bde88d83d', 'request_id': 'req-b495da26-492a-4ed3-8ea8-116493d2837f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.904236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '763e7697890b447c845df94bde88d83d', 'request_id': 'req-b495da26-492a-4ed3-8ea8-116493d2837f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.913552 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '763e7697890b447c845df94bde88d83d', 'request_id': 'req-b495da26-492a-4ed3-8ea8-116493d2837f'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.944470 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] The token cache is being invalidate because project 763e7697890b447c845df94bde88d83d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:30.944765 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.949945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:30.955146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1360/2724] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:30 2022] DELETE /identity/v3/projects/763e7697890b447c845df94bde88d83d => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:30.956475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:30.958057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:30.972889 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16722f1-ea70-42a9-ac09-67d79d6d049e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:31.041343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:31.042266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.042539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.042811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] PATH_INFO: `/v3/users/a5de3972eef34c938b551fe3cdf1d30b` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.045059 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5de3972eef34c938b551fe3cdf1d30b)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:31.047798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:31.113727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5de3972eef34c938b551fe3cdf1d30b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.134815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5de3972eef34c938b551fe3cdf1d30b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.143020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5de3972eef34c938b551fe3cdf1d30b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.143765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5de3972eef34c938b551fe3cdf1d30b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.144383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-1e844318-944e-4844-b5b4-3515b654ddfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5de3972eef34c938b551fe3cdf1d30b'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.163266 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1365/2725] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:30 2022] DELETE /identity/v3/users/a5de3972eef34c938b551fe3cdf1d30b => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:31.388831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:31.390315 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:31.410136 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b495da26-492a-4ed3-8ea8-116493d2837f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:31.472829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:31.495912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:31.496581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.496890 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.497249 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] PATH_INFO: `/v3/projects/1d43731206824351bac01062fa1d0f81` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.498489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d43731206824351bac01062fa1d0f81)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:31.500503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:31.560541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d43731206824351bac01062fa1d0f81', 'request_id': 'req-6428eb3c-d260-4260-8840-966201615823'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.570737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d43731206824351bac01062fa1d0f81', 'request_id': 'req-6428eb3c-d260-4260-8840-966201615823'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.584397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d43731206824351bac01062fa1d0f81', 'request_id': 'req-6428eb3c-d260-4260-8840-966201615823'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.608918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] The token cache is being invalidate because project 1d43731206824351bac01062fa1d0f81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:31.608918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.608918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6428eb3c-d260-4260-8840-966201615823 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:31.608918 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1361/2726] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:31 2022] DELETE /identity/v3/projects/1d43731206824351bac01062fa1d0f81 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:31.613700 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.614181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.614280 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.636782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:31.638041 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:31.713407 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1366/2727] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:31.721917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.722492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.722492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.745606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:31.746999 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:31.756155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1362/2728] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:31.760350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:31.771586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc0b41d-7c29-4dfe-9864-0a2a37ddb0a0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:31.777621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:31.778263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.778535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.779188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.779843 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:31.780373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:31.782040 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:31.795543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1367/2729] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:31.800655 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:31.810028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346600aa-ac29-4d14-9b46-0a68deb9983c None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:31.815753 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:31.816370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.816853 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.817094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.817747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:31.819668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:31.857731 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1363/2730] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:31 2022] POST /identity/v3/projects => generated 375 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:31.861748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:31.875415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4ff2d7-cef0-4cf0-a02c-f5543203cfff admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:31.879816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:31.947390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:31.948573 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:31.948990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:31.949355 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:31.950111 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:31.952159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:31.988050 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1368/2731] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:31 2022] POST /identity/v3/users => generated 288 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:31.992492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.007048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ece8716-50c6-4e0d-9dbc-0ac571e0830b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.016975 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.017628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.017912 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.018185 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.018830 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.020570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.032967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1364/2732] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:32.037541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.048381 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a9f24-04b0-4a02-8e62-4a39ddc89774 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.072566 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710960c9-7ca7-4fd7-a9f3-388e433009a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.073522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710960c9-7ca7-4fd7-a9f3-388e433009a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.074067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710960c9-7ca7-4fd7-a9f3-388e433009a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.074282 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710960c9-7ca7-4fd7-a9f3-388e433009a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.074980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710960c9-7ca7-4fd7-a9f3-388e433009a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.076708 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710960c9-7ca7-4fd7-a9f3-388e433009a9 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.088932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1369/2733] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:32.093512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.104664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9f160b-e18a-4e89-95b6-c046f93bf718 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.109388 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.110038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.110316 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.110589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] PATH_INFO: `/v3/projects/c9a1a86878fe49b780b7988142c72bd0/users/d63286df5e5b44a78ca2c0c251975cc5/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.127172 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9a1a86878fe49b780b7988142c72bd0, user_id=d63286df5e5b44a78ca2c0c251975cc5, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.129862 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.148694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1365/2734] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:32 2022] PUT /identity/v3/projects/c9a1a86878fe49b780b7988142c72bd0/users/d63286df5e5b44a78ca2c0c251975cc5/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:32.161410 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.161821 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.162254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.192770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] MFA Rules not processed for user `d63286df5e5b44a78ca2c0c251975cc5`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:32.194302 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:32.286249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1370/2735] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:32 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:32.290895 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.374100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27493b81-603f-4c7b-8314-e4f940f81625 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.380205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.380816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.381166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.381464 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.382109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.384034 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.401474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:32.420920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1366/2736] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:32 2022] POST /identity/v3/projects => generated 377 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:18:32.423754 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.435317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb5639-6cb1-4aff-bd4c-de4a94e47fa9 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.507463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.508353 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.508793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.509307 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.510105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.512184 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.543103 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1371/2737] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:32 2022] POST /identity/v3/users => generated 289 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:32.547535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.559646 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e7137-e2b4-48b5-9e8a-e35ca5bebc77 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.567955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.567955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.567955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.567955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.568902 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.570547 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.589632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.590883 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1367/2738] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:32.597092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:32.601770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80bb542-24ab-459a-af8e-025eb992b87c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.611830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c15cb06-75b7-40ee-9e89-bebaf638bdbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.612456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c15cb06-75b7-40ee-9e89-bebaf638bdbe admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.613393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c15cb06-75b7-40ee-9e89-bebaf638bdbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.613685 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c15cb06-75b7-40ee-9e89-bebaf638bdbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.614322 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c15cb06-75b7-40ee-9e89-bebaf638bdbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.615927 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c15cb06-75b7-40ee-9e89-bebaf638bdbe admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.629056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1372/2739] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:18:32.633129 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.646199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a280578-ba2e-4ed6-b81d-62f084eeaff2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.651360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.652015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.652295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.652570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] PATH_INFO: `/v3/projects/3c9f37deef97437383ce932ceaf96138/users/c8f9605e1255411da2162e8e80b616e6/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.672498 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c9f37deef97437383ce932ceaf96138, user_id=c8f9605e1255411da2162e8e80b616e6, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.674680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.690802 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1368/2740] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:32 2022] PUT /identity/v3/projects/3c9f37deef97437383ce932ceaf96138/users/c8f9605e1255411da2162e8e80b616e6/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:32.703549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.703837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.704106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.738495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] MFA Rules not processed for user `c8f9605e1255411da2162e8e80b616e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:32.739900 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:32.813267 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1373/2741] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:32.818743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.896533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cb251e-cfaf-449b-9d4b-addebd5e86bb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:32.902107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:32.902831 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:32.903205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:32.903545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:32.904256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:32.906270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:32.942060 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:32.947784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1369/2742] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:32 2022] POST /identity/v3/projects => generated 377 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:32.953770 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044565ed-36ea-4526-aeba-30ce69c7d107 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:33.023133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:33.023916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.024303 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.024678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.025581 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:33.027597 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:33.051138 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1374/2743] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:32 2022] POST /identity/v3/users => generated 295 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:33.055471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:33.074709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd46fb8-c9cc-4533-8056-97783041a77e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:33.079348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:33.079952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.080230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.080505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.081177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:33.082836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:33.096471 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1370/2744] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:33.100896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:33.114881 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad0bd6e-d511-48cf-aa06-b87f47f40dcf admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:33.120949 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:33.121748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.122214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.123319 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] PATH_INFO: `/v3/projects/f4d36da53f7f4a32a5eb7fa23c4467b6/users/5bcccc438e554b4086566c03cc173631/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.142707 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4d36da53f7f4a32a5eb7fa23c4467b6, user_id=5bcccc438e554b4086566c03cc173631, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:33.145719 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:33.163389 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1375/2745] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:33 2022] PUT /identity/v3/projects/f4d36da53f7f4a32a5eb7fa23c4467b6/users/5bcccc438e554b4086566c03cc173631/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:33.167386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:33.178531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e22a219-d341-4b60-9ee3-f1e3fc9c592d admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:33.252582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b09690f-d9bd-4986-acbf-3b53778a54cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:33.254225 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09690f-d9bd-4986-acbf-3b53778a54cc admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.254517 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09690f-d9bd-4986-acbf-3b53778a54cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.255043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b09690f-d9bd-4986-acbf-3b53778a54cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.256009 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09690f-d9bd-4986-acbf-3b53778a54cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:33.258102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b09690f-d9bd-4986-acbf-3b53778a54cc admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:33.274492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1371/2746] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:33.276220 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:33.294824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0450997-cd97-43f3-ace5-2802407dcc9c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:33.312896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:33.312896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.312896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.312896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] PATH_INFO: `/v3/projects/f4d36da53f7f4a32a5eb7fa23c4467b6/users/5bcccc438e554b4086566c03cc173631/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.313506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4d36da53f7f4a32a5eb7fa23c4467b6, user_id=5bcccc438e554b4086566c03cc173631, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:33.313506 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:33.330543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1376/2747] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:33 2022] PUT /identity/v3/projects/f4d36da53f7f4a32a5eb7fa23c4467b6/users/5bcccc438e554b4086566c03cc173631/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:33.340806 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.341223 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.341608 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.375198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] MFA Rules not processed for user `5bcccc438e554b4086566c03cc173631`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:33.375719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:33.397681 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:33.460781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1372/2748] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:33 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:33.473845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:33.475704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:33.646598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:33.670077 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00385a5-d95e-4809-8ed9-1808b5669caa admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:33.674713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:33.675326 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:33.675610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:33.675871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:33.739930 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:33.741549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:33.760998 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1377/2749] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:35.343112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:35.344534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:35.357099 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030724aa-9c86-4195-a962-d13349a21dd3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:35.361457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:35.362056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:35.362321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:35.362586 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:35.429133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:35.430661 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:35.440672 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1373/2750] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:36.115113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:36.125181 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ea51f6-cd0b-4e1e-8191-91e3dbdad776 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:36.129811 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:36.130441 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:36.130706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:36.130963 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:36.140605 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:36.142372 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:36.152888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1378/2751] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:36 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:40.043944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:40.053045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafb96a9-4dcf-4f8b-a543-7b977797f13e service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:40.056590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:40.057218 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:40.057499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:40.057768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:40.059466 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:40.117188 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:40.118611 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d4f50f-e678-43e5-b4b5-bb15ea559904 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:40.128043 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1374/2752] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:40 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:40.202792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:40.269768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4cdb59-1ad2-4c1c-a47b-a565321eebf0 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:40.274278 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:40.274898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:40.275207 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:40.275469 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] PATH_INFO: `/v3/users/d63286df5e5b44a78ca2c0c251975cc5` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:40.276593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d63286df5e5b44a78ca2c0c251975cc5)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:40.278545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:40.331409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd63286df5e5b44a78ca2c0c251975cc5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.343679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd63286df5e5b44a78ca2c0c251975cc5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.350405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd63286df5e5b44a78ca2c0c251975cc5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.351082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd63286df5e5b44a78ca2c0c251975cc5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.351673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd63286df5e5b44a78ca2c0c251975cc5'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.355238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:40.370553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1379/2753] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:40 2022] DELETE /identity/v3/users/d63286df5e5b44a78ca2c0c251975cc5 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:40.372154 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:40.372438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:40.372707 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:40.392067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:40.393937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:40.437930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:40.453130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1375/2754] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:40.463263 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:40.464713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:40.521932 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd0b904-6e92-4361-898f-f7a19d6f8a66 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:40.525583 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:40.526183 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:40.526464 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:40.526736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:40.536471 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:40.537916 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:40.547129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1380/2755] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:40.806905 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:40.808379 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:40.820833 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5828fa-4286-461c-acff-4587d71e88c2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:40.825534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:40.826206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:40.826467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:40.826738 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] PATH_INFO: `/v3/projects/c9a1a86878fe49b780b7988142c72bd0` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:40.827874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9a1a86878fe49b780b7988142c72bd0)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:40.829990 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:40.875956 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9a1a86878fe49b780b7988142c72bd0', 'request_id': 'req-71ba0490-4ae3-4436-bb2a-fbed278f85f8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.883754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9a1a86878fe49b780b7988142c72bd0', 'request_id': 'req-71ba0490-4ae3-4436-bb2a-fbed278f85f8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.889946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9a1a86878fe49b780b7988142c72bd0', 'request_id': 'req-71ba0490-4ae3-4436-bb2a-fbed278f85f8'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.900102 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] The token cache is being invalidate because project c9a1a86878fe49b780b7988142c72bd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:40.900391 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.901057 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:40.902508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1376/2756] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:40 2022] DELETE /identity/v3/projects/c9a1a86878fe49b780b7988142c72bd0 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:40.906519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:40.916714 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b137319-a84b-4220-aef3-583738104b21 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:40.971943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:40.972555 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:40.972830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:40.973120 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] PATH_INFO: `/v3/users/c8f9605e1255411da2162e8e80b616e6` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:40.974237 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8f9605e1255411da2162e8e80b616e6)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:40.976123 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:41.018967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8f9605e1255411da2162e8e80b616e6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.030686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8f9605e1255411da2162e8e80b616e6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.038658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8f9605e1255411da2162e8e80b616e6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.039340 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8f9605e1255411da2162e8e80b616e6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.039958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8f9605e1255411da2162e8e80b616e6'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.052414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1381/2757] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:40 2022] DELETE /identity/v3/users/c8f9605e1255411da2162e8e80b616e6 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:41.318965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:41.320382 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:41.337747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ba0490-4ae3-4436-bb2a-fbed278f85f8 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:41.406252 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:41.406898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:41.407228 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:41.407500 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] PATH_INFO: `/v3/projects/3c9f37deef97437383ce932ceaf96138` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:41.408803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c9f37deef97437383ce932ceaf96138)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:41.410829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:41.458659 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c9f37deef97437383ce932ceaf96138', 'request_id': 'req-f0e9bca5-d912-410a-82fa-c00e05be007b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.467158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c9f37deef97437383ce932ceaf96138', 'request_id': 'req-f0e9bca5-d912-410a-82fa-c00e05be007b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.474302 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c9f37deef97437383ce932ceaf96138', 'request_id': 'req-f0e9bca5-d912-410a-82fa-c00e05be007b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.485484 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] The token cache is being invalidate because project 3c9f37deef97437383ce932ceaf96138 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:41.485800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.486467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.487972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1377/2758] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:41 2022] DELETE /identity/v3/projects/3c9f37deef97437383ce932ceaf96138 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:41.491951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:41.493628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:41.506075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5122d36-9b45-4539-9457-bdc0693ceeb3 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:41.518787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:41.567958 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:41.568609 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:41.568980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:41.569313 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] PATH_INFO: `/v3/users/5bcccc438e554b4086566c03cc173631` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:41.570517 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bcccc438e554b4086566c03cc173631)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:41.572553 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:41.623157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bcccc438e554b4086566c03cc173631'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.637029 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bcccc438e554b4086566c03cc173631'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.644400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bcccc438e554b4086566c03cc173631'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.645127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bcccc438e554b4086566c03cc173631'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.645729 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6788d0bf-9492-4e95-98ce-611b3f0440a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bcccc438e554b4086566c03cc173631'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:41.659190 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1382/2759] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:18:41 2022] DELETE /identity/v3/users/5bcccc438e554b4086566c03cc173631 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:41.938044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:41.939928 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:41.951609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:41.955942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e9bca5-d912-410a-82fa-c00e05be007b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.016507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.017748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.018026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.018294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] PATH_INFO: `/v3/projects/f4d36da53f7f4a32a5eb7fa23c4467b6` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.019527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4d36da53f7f4a32a5eb7fa23c4467b6)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.021621 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.068189 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4d36da53f7f4a32a5eb7fa23c4467b6', 'request_id': 'req-f20bf1dd-32e8-400e-9640-184eb68e059b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:42.075640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4d36da53f7f4a32a5eb7fa23c4467b6', 'request_id': 'req-f20bf1dd-32e8-400e-9640-184eb68e059b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:42.081576 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4d36da53f7f4a32a5eb7fa23c4467b6', 'request_id': 'req-f20bf1dd-32e8-400e-9640-184eb68e059b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:42.091530 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] The token cache is being invalidate because project f4d36da53f7f4a32a5eb7fa23c4467b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:18:42.091813 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:42.092428 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-f20bf1dd-32e8-400e-9640-184eb68e059b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:18:42.093894 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1378/2760] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:18:41 2022] DELETE /identity/v3/projects/f4d36da53f7f4a32a5eb7fa23c4467b6 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:18:42.098946 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.099254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.099534 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.121159 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:42.122785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:42.182478 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1383/2761] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:42.190930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.191295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.191643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.211932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:42.213269 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:42.221328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1379/2762] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:42.226289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.237646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5ceae7-b8d1-4f3e-8bf4-ab5ddefa1683 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.243052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.243800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.244087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.244357 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.245069 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:18:42.245370 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.246954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.263896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1384/2763] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:18:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:18:42.265546 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.279385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc331c7-0dc4-4414-8074-e21c2974a41b None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.285075 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.285694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.285972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.286241 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.286873 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.288758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.324818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1380/2764] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:18:42 2022] POST /identity/v3/projects => generated 373 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:42.329093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.339283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46612bff-4952-4901-8ac9-6e0ebdfcced0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.359709 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:42.401756 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.402400 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.402683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.403091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.403726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.405690 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.426763 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1385/2765] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:18:42 2022] POST /identity/v3/users => generated 287 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:18:42.430942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.441828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483d63e-52b2-44a7-998b-51993cd0aeec admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.446419 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.447010 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.447287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.447554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.448184 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.449843 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.460166 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1381/2766] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:42.463888 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.474107 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9443520a-a069-414e-aa3a-8319b7a82dee admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.478838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445980e-686b-4127-a5c0-0b55df977f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.479422 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445980e-686b-4127-a5c0-0b55df977f78 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.479706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445980e-686b-4127-a5c0-0b55df977f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.479981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445980e-686b-4127-a5c0-0b55df977f78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.480645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1445980e-686b-4127-a5c0-0b55df977f78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.482274 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1445980e-686b-4127-a5c0-0b55df977f78 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.492910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1386/2767] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:18:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:18:42.496963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.507087 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddcf1ee-27e3-4a2d-9db2-6afedd15e7dc admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.511692 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.512322 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.512601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.512884 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] PATH_INFO: `/v3/projects/5fa52688e9da47babb3d9002e7ec137a/users/55d8b5f167e74f32b78ac2aa0debe7db/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.527499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fa52688e9da47babb3d9002e7ec137a, user_id=55d8b5f167e74f32b78ac2aa0debe7db, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.530133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.542893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1382/2768] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:18:42 2022] PUT /identity/v3/projects/5fa52688e9da47babb3d9002e7ec137a/users/55d8b5f167e74f32b78ac2aa0debe7db/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:18:42.547832 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737762d-e252-4902-a2d0-bc4b2b4d82e2 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.548115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737762d-e252-4902-a2d0-bc4b2b4d82e2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.548465 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6737762d-e252-4902-a2d0-bc4b2b4d82e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.567535 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-6737762d-e252-4902-a2d0-bc4b2b4d82e2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:42.568802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6737762d-e252-4902-a2d0-bc4b2b4d82e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:42.627228 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1387/2769] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:42.637181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:42.638951 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:42.673174 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:42.698255 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a2eb3-098d-4e88-8583-4b891530afc4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:42.701881 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:42.702480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:42.702757 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:42.703028 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:42.712088 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:42.713601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:42.723424 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1383/2770] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:43.453727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e93a16-7df6-47e3-98c4-8a6d6c5d021a None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:43.454010 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e93a16-7df6-47e3-98c4-8a6d6c5d021a None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:43.454281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e93a16-7df6-47e3-98c4-8a6d6c5d021a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:43.483247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-01e93a16-7df6-47e3-98c4-8a6d6c5d021a None None] MFA Rules not processed for user `55d8b5f167e74f32b78ac2aa0debe7db`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:43.484545 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-01e93a16-7df6-47e3-98c4-8a6d6c5d021a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:43.559275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1388/2771] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:43.571199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:43.573474 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:43.642837 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d144bb-cd78-4963-b09d-8c2424430e07 service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:43.647452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abd77f4-2218-4d05-b53a-974d9602fc26 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:43.648062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abd77f4-2218-4d05-b53a-974d9602fc26 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:43.648439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abd77f4-2218-4d05-b53a-974d9602fc26 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:43.649040 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abd77f4-2218-4d05-b53a-974d9602fc26 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:43.659917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abd77f4-2218-4d05-b53a-974d9602fc26 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:43.661465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abd77f4-2218-4d05-b53a-974d9602fc26 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:43.671607 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1384/2772] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:43 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:44.986504 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d19869-0c44-4f87-af03-b879d136b300 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:44.987051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d19869-0c44-4f87-af03-b879d136b300 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:44.987051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d19869-0c44-4f87-af03-b879d136b300 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:44.988408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1389/2773] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:18:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:18:44.995003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c632d-c74d-4f92-b48a-70a4495bbdb3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:44.995443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c632d-c74d-4f92-b48a-70a4495bbdb3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:44.995702 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c632d-c74d-4f92-b48a-70a4495bbdb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.018392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-8f6c632d-c74d-4f92-b48a-70a4495bbdb3 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:45.019628 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-8f6c632d-c74d-4f92-b48a-70a4495bbdb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:45.094507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1385/2774] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:18:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:45.106245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d19869-0c44-4f87-af03-b879d136b300 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:45.108066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-48d19869-0c44-4f87-af03-b879d136b300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:45.151000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:45.172724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d19869-0c44-4f87-af03-b879d136b300 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:45.176815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48773718-f48f-40bd-9cb1-f2e37f603d9e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:45.177443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48773718-f48f-40bd-9cb1-f2e37f603d9e service glance] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.177727 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48773718-f48f-40bd-9cb1-f2e37f603d9e service glance] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.178004 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48773718-f48f-40bd-9cb1-f2e37f603d9e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.187873 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48773718-f48f-40bd-9cb1-f2e37f603d9e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:45.189956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48773718-f48f-40bd-9cb1-f2e37f603d9e service glance] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:45.200447 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1390/2775] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:45.267709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48f322-e5c5-4067-828f-fb6605c3f3f1 None None] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.267994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48f322-e5c5-4067-828f-fb6605c3f3f1 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.268268 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48f322-e5c5-4067-828f-fb6605c3f3f1 None None] PATH_INFO: `/` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.269622 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1386/2776] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:18:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:18:45.274393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd71b62-ac0f-42d5-839d-fc72e3c14a39 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.274682 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd71b62-ac0f-42d5-839d-fc72e3c14a39 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.274980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd71b62-ac0f-42d5-839d-fc72e3c14a39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.294221 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-9fd71b62-ac0f-42d5-839d-fc72e3c14a39 None None] MFA Rules not processed for user `afdc21c25f734a51aa520e7d4e53455b`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:45.295430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-9fd71b62-ac0f-42d5-839d-fc72e3c14a39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:45.303430 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1391/2777] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:18:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:45.317127 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b48f322-e5c5-4067-828f-fb6605c3f3f1 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:45.329649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b48f322-e5c5-4067-828f-fb6605c3f3f1 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:45.334170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d275c286-c36c-4179-85e5-05ac490e4e61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:45.334829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275c286-c36c-4179-85e5-05ac490e4e61 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.335175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275c286-c36c-4179-85e5-05ac490e4e61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.335587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275c286-c36c-4179-85e5-05ac490e4e61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.343264 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d275c286-c36c-4179-85e5-05ac490e4e61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:45.344684 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d275c286-c36c-4179-85e5-05ac490e4e61 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:45.352406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1387/2778] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:45.436576 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f6d9d-e019-415e-bbbe-ac778d5342d2 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.436862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f6d9d-e019-415e-bbbe-ac778d5342d2 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.437158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f6d9d-e019-415e-bbbe-ac778d5342d2 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.438493 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1392/2779] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:18:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:18:45.444035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b2e96b-caad-4139-86ca-cf2374ae2104 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.444326 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b2e96b-caad-4139-86ca-cf2374ae2104 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.444604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b2e96b-caad-4139-86ca-cf2374ae2104 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.463878 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-81b2e96b-caad-4139-86ca-cf2374ae2104 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:45.465116 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-81b2e96b-caad-4139-86ca-cf2374ae2104 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:45.474751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1388/2780] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:18:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:45.485088 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f6d9d-e019-415e-bbbe-ac778d5342d2 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:45.496980 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f6d9d-e019-415e-bbbe-ac778d5342d2 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:45.500502 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d3aecb-786e-4b22-9475-54c0b27e4e83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:45.501115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3aecb-786e-4b22-9475-54c0b27e4e83 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:45.501456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3aecb-786e-4b22-9475-54c0b27e4e83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:45.501723 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3aecb-786e-4b22-9475-54c0b27e4e83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:45.512538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d3aecb-786e-4b22-9475-54c0b27e4e83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:45.514129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d3aecb-786e-4b22-9475-54c0b27e4e83 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:45.525893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1393/2781] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:46.060110 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:46.060594 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:46.060714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:46.085867 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] MFA Rules not processed for user `b38384d843d04503b69e5cb61af6c6ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:46.087242 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:46.129242 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ce58d0-c680-4bcd-b608-9afe275fbe75 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:46.129668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ce58d0-c680-4bcd-b608-9afe275fbe75 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:46.129794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ce58d0-c680-4bcd-b608-9afe275fbe75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:46.150537 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-79ce58d0-c680-4bcd-b608-9afe275fbe75 None None] MFA Rules not processed for user `55d8b5f167e74f32b78ac2aa0debe7db`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:46.151996 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-79ce58d0-c680-4bcd-b608-9afe275fbe75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:46.164956 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1394/2782] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:46.179688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:46.192526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1389/2783] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:18:46 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:48.196659 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e87d5c8-0be6-442b-a1ba-474f20457236 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:48.197174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e87d5c8-0be6-442b-a1ba-474f20457236 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:48.197275 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e87d5c8-0be6-442b-a1ba-474f20457236 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:48.217668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-8e87d5c8-0be6-442b-a1ba-474f20457236 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:48.218905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8e87d5c8-0be6-442b-a1ba-474f20457236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:48.303149 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1395/2784] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:18:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:48.319431 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:48.321437 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:48.404675 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd887f2-06b1-42c7-bb09-d2ab7fd976b6 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:48.417822 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:48.418465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:48.418745 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:48.419020 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:48.428804 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:48.430420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:48.439888 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1390/2785] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:18:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:18:58.292664 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=48 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:58.435493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=35 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:10.147544 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d785be-ea56-4a39-968d-da872414657c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:10.148427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d785be-ea56-4a39-968d-da872414657c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:10.148806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d785be-ea56-4a39-968d-da872414657c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:10.173716 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-97d785be-ea56-4a39-968d-da872414657c None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:10.175072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-97d785be-ea56-4a39-968d-da872414657c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:10.182308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1396/2786] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 29 02:19:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:10.196202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:10.206793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef1044-f2df-4152-ad08-9ddafe5c60fb service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:10.211776 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:10.212366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:10.212643 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:10.213031 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:10.221919 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:10.223593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:10.231960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1391/2787] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:19:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:19:11.437724 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:11.438203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:11.438300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:11.457289 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] MFA Rules not processed for user `55d8b5f167e74f32b78ac2aa0debe7db`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:11.458492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:11.465415 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1397/2788] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:19:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:14.648803 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:14.658735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2fc9f4-eb7b-4cf4-9821-cdb7d4924159 service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:14.662714 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:14.663308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:14.663587 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:14.663855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:14.665699 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:14.735847 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:14.738212 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:14.750084 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1392/2789] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:14 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:21.451568 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:23.701819 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:23.722498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a92d29-33e4-480f-87d0-7047b0b7f021 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:23.726186 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:23.726807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:23.727082 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:23.727396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:23.729168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:23.790383 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:23.791815 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:23.810725 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1398/2790] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:24.743782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=36 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:30.051539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:30.053144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:30.105247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a6df6-a69f-4868-adda-e482f11649da service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:30.110598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:30.111233 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:30.111513 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:30.111791 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] PATH_INFO: `/v3/users/ef9f29b86ed947919bcb5d6166ac0641` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:30.112970 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef9f29b86ed947919bcb5d6166ac0641)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:30.114920 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:30.170234 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef9f29b86ed947919bcb5d6166ac0641'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.178333 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef9f29b86ed947919bcb5d6166ac0641'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.179123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef9f29b86ed947919bcb5d6166ac0641'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.179807 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef9f29b86ed947919bcb5d6166ac0641'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.191236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef9f29b86ed947919bcb5d6166ac0641'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.216170 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1393/2791] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:19:30 2022] DELETE /identity/v3/users/ef9f29b86ed947919bcb5d6166ac0641 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:30.386418 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:30.396694 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a89261-0ddd-453d-bb8c-4f4ff212d057 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:30.454008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:30.454648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:30.454931 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:30.455208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] PATH_INFO: `/v3/projects/18eeabc669304148930130efa1737cbb` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:30.456336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=18eeabc669304148930130efa1737cbb)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:30.458328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:30.503249 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18eeabc669304148930130efa1737cbb', 'request_id': 'req-82831841-811b-4df2-8949-f8a50dd26a2d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.509637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18eeabc669304148930130efa1737cbb', 'request_id': 'req-82831841-811b-4df2-8949-f8a50dd26a2d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.516090 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18eeabc669304148930130efa1737cbb', 'request_id': 'req-82831841-811b-4df2-8949-f8a50dd26a2d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.527243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] The token cache is being invalidate because project 18eeabc669304148930130efa1737cbb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:19:30.527529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.528187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.529711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1399/2792] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:19:30 2022] DELETE /identity/v3/projects/18eeabc669304148930130efa1737cbb => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:30.534243 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:30.535712 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:30.549407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d8e61b-30ac-4c47-a32a-9237be5aa050 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:30.604205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:30.604829 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:30.605130 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:30.605452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] PATH_INFO: `/v3/users/854f55ef9cc44cf687a17f1de857ff6b` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:30.606640 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=854f55ef9cc44cf687a17f1de857ff6b)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:30.608748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:30.649486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:30.656605 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '854f55ef9cc44cf687a17f1de857ff6b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.664529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '854f55ef9cc44cf687a17f1de857ff6b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.665301 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '854f55ef9cc44cf687a17f1de857ff6b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.665898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '854f55ef9cc44cf687a17f1de857ff6b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.673533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '854f55ef9cc44cf687a17f1de857ff6b'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.690688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1394/2793] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:19:30 2022] DELETE /identity/v3/users/854f55ef9cc44cf687a17f1de857ff6b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:30.868199 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:30.869851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:30.882687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82831841-811b-4df2-8949-f8a50dd26a2d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:30.887525 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:30.942349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:30.942977 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:30.943258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:30.943531 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] PATH_INFO: `/v3/projects/1af0fde6012a448dbb0a35db223f8aaa` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:30.944646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1af0fde6012a448dbb0a35db223f8aaa)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:30.946585 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:30.990858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1af0fde6012a448dbb0a35db223f8aaa', 'request_id': 'req-641432ee-4ec4-4f95-8857-1db6aae1d367'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:30.997839 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1af0fde6012a448dbb0a35db223f8aaa', 'request_id': 'req-641432ee-4ec4-4f95-8857-1db6aae1d367'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:31.003317 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1af0fde6012a448dbb0a35db223f8aaa', 'request_id': 'req-641432ee-4ec4-4f95-8857-1db6aae1d367'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:31.013912 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] The token cache is being invalidate because project 1af0fde6012a448dbb0a35db223f8aaa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:19:31.014198 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:31.014833 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-641432ee-4ec4-4f95-8857-1db6aae1d367 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:31.016818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1400/2794] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:19:30 2022] DELETE /identity/v3/projects/1af0fde6012a448dbb0a35db223f8aaa => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:31.157446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:31.158944 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:31.268226 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0719a2-65d5-4974-8129-11b9c74deaf0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:31.330206 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:31.330719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:31.330995 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:31.331261 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:31.332989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:31.344893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:31.346443 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:31.355998 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1395/2795] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:32.073432 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf289d2-8fd6-4094-ab65-cca5bbdb56d3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:32.073859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf289d2-8fd6-4094-ab65-cca5bbdb56d3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:32.074201 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf289d2-8fd6-4094-ab65-cca5bbdb56d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:32.105223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-aaf289d2-8fd6-4094-ab65-cca5bbdb56d3 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:32.106423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-aaf289d2-8fd6-4094-ab65-cca5bbdb56d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:32.168479 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1401/2796] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:19:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:19:32.180716 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:32.182128 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:32.210593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:32.254797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7feef0-cad3-4a98-946c-ed102e5020ec service placement] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:32.260008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:32.260615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:32.260926 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:32.261203 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:32.270177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:32.271696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:32.281153 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1396/2797] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:19:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:19:42.162100 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=79,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:42.277152 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=22 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:47.534661 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:47.535140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:47.535246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:47.554554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:47.555746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:47.563314 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1402/2798] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 29 02:19:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:47.575503 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:47.586821 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b659beb4-a5bc-463c-9e84-1a9a5fcd5089 service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:47.591887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:47.592476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:47.592762 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:47.593131 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:47.601729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:47.603290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:47.611273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1397/2799] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:19:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:19:49.537954 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:49.539472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:49.609798 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2242e4c0-a81d-4daf-81c7-f5fa1635dce0 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:49.614281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:49.614907 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:49.615188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:49.615462 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] PATH_INFO: `/v3/users/55d8b5f167e74f32b78ac2aa0debe7db` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:49.616604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55d8b5f167e74f32b78ac2aa0debe7db)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:49.618559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:49.664308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55d8b5f167e74f32b78ac2aa0debe7db'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:49.680746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55d8b5f167e74f32b78ac2aa0debe7db'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:49.690831 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55d8b5f167e74f32b78ac2aa0debe7db'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:49.691642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55d8b5f167e74f32b78ac2aa0debe7db'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:49.692399 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-00886d4a-59be-4e36-aa24-332ea8410844 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55d8b5f167e74f32b78ac2aa0debe7db'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:49.706885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1403/2800] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:19:49 2022] DELETE /identity/v3/users/55d8b5f167e74f32b78ac2aa0debe7db => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:49.892724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:49.923111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3615917-1a37-44d6-a181-873cf54798b5 service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:49.982292 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:49.984360 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:49.984778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:49.985094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] PATH_INFO: `/v3/projects/5fa52688e9da47babb3d9002e7ec137a` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:49.986270 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fa52688e9da47babb3d9002e7ec137a)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:49.988325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.038177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fa52688e9da47babb3d9002e7ec137a', 'request_id': 'req-ef28cad0-9310-45ff-9065-f101028b5fef'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:50.046134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fa52688e9da47babb3d9002e7ec137a', 'request_id': 'req-ef28cad0-9310-45ff-9065-f101028b5fef'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:50.052697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fa52688e9da47babb3d9002e7ec137a', 'request_id': 'req-ef28cad0-9310-45ff-9065-f101028b5fef'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:50.063910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] The token cache is being invalidate because project 5fa52688e9da47babb3d9002e7ec137a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:19:50.064452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:50.065144 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ef28cad0-9310-45ff-9065-f101028b5fef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:50.066723 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1398/2801] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:19:49 2022] DELETE /identity/v3/projects/5fa52688e9da47babb3d9002e7ec137a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:50.071582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.071863 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.072135 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.132115 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:50.133374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:50.170164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:50.196132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1404/2802] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:19:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:19:50.205044 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.205337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.205618 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.226624 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:50.227852 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:50.235828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1399/2803] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:19:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:50.240218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.250369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356ba8bf-f0e0-46f3-9026-109826435b6e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.254950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.255542 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.255818 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.256083 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.256748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:19:50.257073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.258647 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.270871 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1405/2804] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:19:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:19:50.275462 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.284779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b096f8-0d2e-412e-aab0-1891d2987e1f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.289441 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.290038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.290341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.290612 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.291248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.293145 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.322896 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1400/2805] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:19:50 2022] POST /identity/v3/projects => generated 365 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:19:50.327166 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.337571 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7686a7-caa8-46d8-891a-931eb6d1f35e admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.392438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.393066 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.393348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.393620 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.394258 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.396160 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.416384 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1406/2806] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:19:50 2022] POST /identity/v3/users => generated 283 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:19:50.420535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.431385 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c82a7-e9a9-43a1-847a-e3c481be8b35 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.436058 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.436680 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.437522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.437796 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.438477 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.440341 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.450758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1401/2807] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:19:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:19:50.454736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.465063 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28fb37d-8368-496f-b765-da5b4d23259a admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.469793 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3ddfc6-d285-4ee6-aab1-1539c9d12f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.470391 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ddfc6-d285-4ee6-aab1-1539c9d12f33 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.470696 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ddfc6-d285-4ee6-aab1-1539c9d12f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.470969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ddfc6-d285-4ee6-aab1-1539c9d12f33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.471601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3ddfc6-d285-4ee6-aab1-1539c9d12f33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.473223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3ddfc6-d285-4ee6-aab1-1539c9d12f33 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.484290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1407/2808] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:19:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:19:50.488467 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.498347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd275d80-40f3-4ab9-93f8-df73359ef549 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.503535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde091dd-20de-4134-9942-427845980350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.504175 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde091dd-20de-4134-9942-427845980350 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.504457 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde091dd-20de-4134-9942-427845980350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.504730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde091dd-20de-4134-9942-427845980350 admin admin] PATH_INFO: `/v3/projects/ba94b0b77ba94dd9a82812ad8bf60df3/users/804e24f39129438da0ab0aa52bfcae2c/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.518961 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde091dd-20de-4134-9942-427845980350 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba94b0b77ba94dd9a82812ad8bf60df3, user_id=804e24f39129438da0ab0aa52bfcae2c, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.521707 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde091dd-20de-4134-9942-427845980350 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.534328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1402/2809] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:19:50 2022] PUT /identity/v3/projects/ba94b0b77ba94dd9a82812ad8bf60df3/users/804e24f39129438da0ab0aa52bfcae2c/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:50.544336 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7f33ee-885a-4fd8-a983-097a01d1dbe3 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.544836 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7f33ee-885a-4fd8-a983-097a01d1dbe3 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.544988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7f33ee-885a-4fd8-a983-097a01d1dbe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.575557 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-dc7f33ee-885a-4fd8-a983-097a01d1dbe3 None None] MFA Rules not processed for user `804e24f39129438da0ab0aa52bfcae2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:50.576771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-dc7f33ee-885a-4fd8-a983-097a01d1dbe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:50.640475 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1408/2810] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:19:50 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:50.651581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde091dd-20de-4134-9942-427845980350 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:50.653330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-dde091dd-20de-4134-9942-427845980350 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:50.671480 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:50.733590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde091dd-20de-4134-9942-427845980350 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:50.737178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:50.737772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:50.738045 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:50.738311 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:50.746620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:50.748029 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:50.757582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1403/2811] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:50 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:55.630658 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b24203-1edc-47e0-b625-58709a5c3dfb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:55.631092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b24203-1edc-47e0-b625-58709a5c3dfb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:55.631213 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b24203-1edc-47e0-b625-58709a5c3dfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:55.650005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-91b24203-1edc-47e0-b625-58709a5c3dfb None None] MFA Rules not processed for user `e077cbccb35940608bc2224c373c0b99`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:55.651204 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-91b24203-1edc-47e0-b625-58709a5c3dfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:55.682624 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:55.706898 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1409/2812] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:19:55 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:00.753512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=29 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:05.700276 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=8 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:26.088465 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:26.147969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529af797-fc47-4557-954f-5cb7d1975fe0 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:26.152389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:26.153068 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.153349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.153619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] PATH_INFO: `/v3/users/804e24f39129438da0ab0aa52bfcae2c` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.154719 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=804e24f39129438da0ab0aa52bfcae2c)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:26.157123 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:26.201347 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '804e24f39129438da0ab0aa52bfcae2c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.208121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '804e24f39129438da0ab0aa52bfcae2c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.208760 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '804e24f39129438da0ab0aa52bfcae2c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.209386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '804e24f39129438da0ab0aa52bfcae2c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.216411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '804e24f39129438da0ab0aa52bfcae2c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.234258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1404/2813] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:20:26 2022] DELETE /identity/v3/users/804e24f39129438da0ab0aa52bfcae2c => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:26.238669 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.239164 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.239315 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.258569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:26.259759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:26.317450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1410/2814] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:20:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:26.327969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:26.329392 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:26.403600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f14f0c5-25dc-4d31-8795-dc7dbc626a89 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:26.407679 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dbe7be-36c0-4d8f-b8cf-0e485136a222 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:26.408271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dbe7be-36c0-4d8f-b8cf-0e485136a222 service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.408549 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dbe7be-36c0-4d8f-b8cf-0e485136a222 service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.408816 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dbe7be-36c0-4d8f-b8cf-0e485136a222 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.417570 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dbe7be-36c0-4d8f-b8cf-0e485136a222 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:26.418983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dbe7be-36c0-4d8f-b8cf-0e485136a222 service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:26.428117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1405/2815] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:20:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:20:26.694595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:26.696026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:26.708093 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb0f8d7-5dbb-43a1-94c8-9566ecf84692 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:26.712666 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:26.713397 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.713734 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.714006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] PATH_INFO: `/v3/projects/ba94b0b77ba94dd9a82812ad8bf60df3` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.715105 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba94b0b77ba94dd9a82812ad8bf60df3)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:26.717015 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:26.762036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba94b0b77ba94dd9a82812ad8bf60df3', 'request_id': 'req-fe7eeef8-cf64-4775-be41-3a0ff75edc21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.769098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba94b0b77ba94dd9a82812ad8bf60df3', 'request_id': 'req-fe7eeef8-cf64-4775-be41-3a0ff75edc21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.774532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba94b0b77ba94dd9a82812ad8bf60df3', 'request_id': 'req-fe7eeef8-cf64-4775-be41-3a0ff75edc21'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.785872 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] The token cache is being invalidate because project ba94b0b77ba94dd9a82812ad8bf60df3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:20:26.786156 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.786800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-fe7eeef8-cf64-4775-be41-3a0ff75edc21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:26.788256 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1411/2816] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:20:26 2022] DELETE /identity/v3/projects/ba94b0b77ba94dd9a82812ad8bf60df3 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:26.793168 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.793455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.793726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.801729 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:26.816458 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:26.817721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:26.876541 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1406/2817] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:20:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:26.885435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.885748 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.886028 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.906459 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:26.907653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:26.915554 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1412/2818] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:20:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:26.919825 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:26.929271 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7894f9-cde6-448b-9138-fcab08ab5cdc None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:26.934079 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:26.934676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.934953 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.935258 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.935959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:20:26.936259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:26.938401 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:26.949787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1407/2819] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:20:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:20:26.954637 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:26.963529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09ef0e2-d1f2-411e-b919-8d40b9192eba None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:26.968092 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:26.968678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:26.968986 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:26.969253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:26.969874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:26.971743 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:26.997755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1413/2820] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:20:26 2022] POST /identity/v3/projects => generated 383 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:20:27.002446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:27.012122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424cdcee-e493-4f56-9ba2-cbd1dbec05fd admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:27.067537 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:27.068134 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.068418 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.068690 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.069362 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:27.071276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:27.091565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1408/2821] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:20:27 2022] POST /identity/v3/users => generated 292 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:20:27.095753 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:27.105718 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89308c2c-914c-4038-bedc-2865ec355c4d admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:27.110759 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:27.111343 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.111617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.111884 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.112511 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:27.114127 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:27.123601 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1414/2822] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:20:27 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:20:27.127962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:27.137704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2813dcb0-5c0f-40f3-9384-f474a9c7e4d2 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:27.142787 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f42ba-d013-4e83-b58d-50387e58877e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:27.143376 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f42ba-d013-4e83-b58d-50387e58877e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.143660 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f42ba-d013-4e83-b58d-50387e58877e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.143932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f42ba-d013-4e83-b58d-50387e58877e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.144566 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f42ba-d013-4e83-b58d-50387e58877e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:27.146177 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f42ba-d013-4e83-b58d-50387e58877e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:27.157284 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1409/2823] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:20:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:20:27.161179 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:27.170786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a94d6d1-31db-4bb1-afee-34b58ab338b2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:27.175318 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:27.175943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.176216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.176483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] PATH_INFO: `/v3/projects/50e5b07fbfdd4ed190aee4702194a5fa/users/9365ac5a4b274e72b01afca8906ac79a/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.190814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=50e5b07fbfdd4ed190aee4702194a5fa, user_id=9365ac5a4b274e72b01afca8906ac79a, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:27.193494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:27.207026 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1415/2824] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:20:27 2022] PUT /identity/v3/projects/50e5b07fbfdd4ed190aee4702194a5fa/users/9365ac5a4b274e72b01afca8906ac79a/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:27.212038 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75db22d-8d48-45c6-b50f-e53a0010962c None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.212324 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75db22d-8d48-45c6-b50f-e53a0010962c None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.212597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75db22d-8d48-45c6-b50f-e53a0010962c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.231601 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-a75db22d-8d48-45c6-b50f-e53a0010962c None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:27.232784 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a75db22d-8d48-45c6-b50f-e53a0010962c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:27.288987 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1410/2825] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:20:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:27.298579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:27.300259 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:27.326212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:27.366304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a37ca8-fb0a-43ca-a70c-c278bd0b62fa admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:27.369814 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:27.370437 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.370711 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.370979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.380128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:27.381582 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:27.390735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1416/2826] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:20:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:20:28.018406 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de5d6c-ebbf-4870-8bdf-2a6d2ee48df3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:28.018768 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de5d6c-ebbf-4870-8bdf-2a6d2ee48df3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:28.019112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de5d6c-ebbf-4870-8bdf-2a6d2ee48df3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:28.038792 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:28.049793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-33de5d6c-ebbf-4870-8bdf-2a6d2ee48df3 None None] MFA Rules not processed for user `9365ac5a4b274e72b01afca8906ac79a`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:28.050996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-33de5d6c-ebbf-4870-8bdf-2a6d2ee48df3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:28.119593 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1411/2827] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:20:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:28.131206 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:28.132628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:28.191629 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c6cd8d-1116-4923-899e-645e22cbecbe service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:28.195203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c100a-66c3-4d45-a69e-97fe2054b145 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:28.195802 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c100a-66c3-4d45-a69e-97fe2054b145 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:28.196075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c100a-66c3-4d45-a69e-97fe2054b145 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:28.196349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c100a-66c3-4d45-a69e-97fe2054b145 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:28.205030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c100a-66c3-4d45-a69e-97fe2054b145 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:28.206455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c100a-66c3-4d45-a69e-97fe2054b145 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:28.215157 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1417/2828] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:20:28 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:20:29.571561 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:29.572117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:29.572276 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:29.593342 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] MFA Rules not processed for user `9365ac5a4b274e72b01afca8906ac79a`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:29.594590 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:29.601386 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1412/2829] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:20:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:38.211203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=43 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:39.585893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=31 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:15.025990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5477825-3f82-42ef-b829-23589138355c None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:15.026530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5477825-3f82-42ef-b829-23589138355c None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:15.026635 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5477825-3f82-42ef-b829-23589138355c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:15.046378 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-a5477825-3f82-42ef-b829-23589138355c None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:15.047561 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a5477825-3f82-42ef-b829-23589138355c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:15.100567 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1418/2830] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:21:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:15.147366 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:15.148783 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:15.217007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450d897d-3341-4f7b-b49a-626e339b065f None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:15.221848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:15.222455 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:15.222735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:15.223004 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:15.231586 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:15.233178 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:15.241062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1413/2831] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:21:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 29 02:21:15.672726 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5477825-3f82-42ef-b829-23589138355c None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:15.682604 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5477825-3f82-42ef-b829-23589138355c None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:15.686728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:15.687349 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:15.687634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:15.687906 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:15.690073 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:15.703413 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:15.704891 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:15.716388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1419/2832] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:18.377613 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:18.379062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:18.402742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d10fe2-d8fa-4759-91ca-a3ad5215d8af service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:18.406678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:18.407356 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:18.407703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:18.408041 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:18.409982 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:18.412265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:18.424750 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a065e4d8-924d-41b6-83fd-0ca78c4a6bdb service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:18.438406 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:18.439308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:18.439594 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:18.443299 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:18.453730 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:18.547809 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:18.549630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:18.575140 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1414/2833] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:18.580632 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:18.582643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:18.600364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1420/2834] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:23.980972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:23.982404 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:24.019364 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b16724-09fd-42b0-bac5-4156bf3e8e4a service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:24.034940 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:24.039231 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:24.039751 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:24.040925 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] PATH_INFO: `/v3/users/b38384d843d04503b69e5cb61af6c6ec` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:24.043627 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b38384d843d04503b69e5cb61af6c6ec)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:24.045959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:24.130631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b38384d843d04503b69e5cb61af6c6ec'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.134439 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:24.143259 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b38384d843d04503b69e5cb61af6c6ec'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.144389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b38384d843d04503b69e5cb61af6c6ec'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.145330 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b38384d843d04503b69e5cb61af6c6ec'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.160698 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b38384d843d04503b69e5cb61af6c6ec'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.196487 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1415/2835] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:23 2022] DELETE /identity/v3/users/b38384d843d04503b69e5cb61af6c6ec => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:24.398023 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:24.411451 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f25138-bedc-41a0-9038-884f0d3e0726 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:24.481279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:24.482182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:24.482732 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:24.483224 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] PATH_INFO: `/v3/projects/165a7d71a21240a79e77abbe80717389` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:24.484362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] RBAC: Authorizing `identity:delete_project(project_id=165a7d71a21240a79e77abbe80717389)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:24.486425 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:24.514124 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=100 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:24.542262 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '165a7d71a21240a79e77abbe80717389', 'request_id': 'req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.554603 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '165a7d71a21240a79e77abbe80717389', 'request_id': 'req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.562361 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '165a7d71a21240a79e77abbe80717389', 'request_id': 'req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.580987 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] The token cache is being invalidate because project 165a7d71a21240a79e77abbe80717389 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:24.581645 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.582505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.584529 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1421/2836] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:24 2022] DELETE /identity/v3/projects/165a7d71a21240a79e77abbe80717389 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:24.588652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:24.590433 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:24.606516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1132cd6e-4c28-493f-bb88-b8b473663386 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:24.673721 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:24.674420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:24.674697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:24.674972 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] PATH_INFO: `/v3/users/96c7129b47e94199bd8975bd3d3c98c7` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:24.676308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=96c7129b47e94199bd8975bd3d3c98c7)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:24.678272 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:24.738945 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96c7129b47e94199bd8975bd3d3c98c7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.752694 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96c7129b47e94199bd8975bd3d3c98c7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.753893 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96c7129b47e94199bd8975bd3d3c98c7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.754737 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96c7129b47e94199bd8975bd3d3c98c7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.765378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-64c5ef90-3df2-4376-9b2a-34452cace8fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96c7129b47e94199bd8975bd3d3c98c7'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:24.784209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1416/2837] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:24 2022] DELETE /identity/v3/users/96c7129b47e94199bd8975bd3d3c98c7 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:24.993955 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:24.995910 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:25.010055 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb03bfa-5a20-4510-9bf9-9caf0d86e317 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:25.088189 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:25.088834 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:25.089165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:25.089438 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] PATH_INFO: `/v3/projects/47aa1e9c32034f62b9c720cc48ec7338` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:25.090586 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47aa1e9c32034f62b9c720cc48ec7338)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:25.092649 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:25.149151 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47aa1e9c32034f62b9c720cc48ec7338', 'request_id': 'req-40677952-e97d-4220-82c5-a4f5b20fbe90'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.157592 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47aa1e9c32034f62b9c720cc48ec7338', 'request_id': 'req-40677952-e97d-4220-82c5-a4f5b20fbe90'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.165768 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47aa1e9c32034f62b9c720cc48ec7338', 'request_id': 'req-40677952-e97d-4220-82c5-a4f5b20fbe90'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.179997 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] The token cache is being invalidate because project 47aa1e9c32034f62b9c720cc48ec7338 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:25.180380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.181178 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-40677952-e97d-4220-82c5-a4f5b20fbe90 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.183283 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1422/2838] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:24 2022] DELETE /identity/v3/projects/47aa1e9c32034f62b9c720cc48ec7338 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:30.436790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1808b80-a31b-4c13-a607-238fddba18a3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:30.437349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1808b80-a31b-4c13-a607-238fddba18a3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:30.437349 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1808b80-a31b-4c13-a607-238fddba18a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:30.456283 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f1808b80-a31b-4c13-a607-238fddba18a3 None None] MFA Rules not processed for user `9365ac5a4b274e72b01afca8906ac79a`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:30.457519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f1808b80-a31b-4c13-a607-238fddba18a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:30.497897 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:30.513649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1417/2839] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:21:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:32.313232 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d4cb51-2e02-4850-b660-9475a338dca5 None None] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.313810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d4cb51-2e02-4850-b660-9475a338dca5 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.313810 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d4cb51-2e02-4850-b660-9475a338dca5 None None] PATH_INFO: `/` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.315543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1423/2840] 10.222.0.27 () {54 vars in 921 bytes} [Thu Sep 29 02:21:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:21:32.322345 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba18d067-e5ed-4e25-937c-ff3f394c02b8 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.322863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba18d067-e5ed-4e25-937c-ff3f394c02b8 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.322863 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba18d067-e5ed-4e25-937c-ff3f394c02b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.361811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ba18d067-e5ed-4e25-937c-ff3f394c02b8 None None] MFA Rules not processed for user `e528956ee4704ac0b5ce4c62826b71ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:32.363125 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ba18d067-e5ed-4e25-937c-ff3f394c02b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:32.437321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1418/2841] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Sep 29 02:21:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:32.449549 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d4cb51-2e02-4850-b660-9475a338dca5 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:32.451598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b2d4cb51-2e02-4850-b660-9475a338dca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:32.497792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:32.521374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d4cb51-2e02-4850-b660-9475a338dca5 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:32.526671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0298fb95ae9341c6a1eb8da546d93cb2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:32.526671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.526945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.527143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.538838 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:32.540966 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:32.561824 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1424/2842] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:35.725287 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.725790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.725899 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.777623 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:35.778800 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:35.846018 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1419/2843] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Sep 29 02:21:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:35.858461 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:35.859859 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:35.925874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d248be-4f18-45d7-b0bd-b8e1f8436d0e service glance] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:35.930735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9fd257-fb6c-4cad-9e37-8d9d0c5f3803 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:35.931321 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9fd257-fb6c-4cad-9e37-8d9d0c5f3803 service swift] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.931595 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9fd257-fb6c-4cad-9e37-8d9d0c5f3803 service swift] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.931862 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9fd257-fb6c-4cad-9e37-8d9d0c5f3803 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.939788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9fd257-fb6c-4cad-9e37-8d9d0c5f3803 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.941377 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9fd257-fb6c-4cad-9e37-8d9d0c5f3803 service swift] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.949673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1425/2844] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:21:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:21:44.798839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:44.800317 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:44.844340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:44.855600 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dc917f-0677-417d-a9d7-5ad5d51d2812 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:44.859246 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:44.859845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:44.860121 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:44.860389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:44.862081 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:44.871450 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:44.872855 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:44.880887 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1420/2845] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:45.946113 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=41 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:50.320514 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:50.321640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:50.321640 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:50.341408 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] MFA Rules not processed for user `3546ff5fc9094d35921920f32e0c7476`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:50.342532 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:50.349820 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1426/2846] 10.222.0.27 () {58 vars in 967 bytes} [Thu Sep 29 02:21:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:50.362295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:50.383674 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3c996-4d83-4957-ae6c-47b0fbbe85eb service neutron] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:50.388801 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '10649368308f4dc4b8b887319cbcac8f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:50.389446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:50.389756 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:50.390032 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:50.398695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:50.400850 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:50.409200 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1421/2847] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Sep 29 02:21:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:22:00.335139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:00.405706 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=17 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:07.434786 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:07.436416 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:07.453325 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3695bc52-a4e2-4d5e-9502-062f9a451810 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:07.459398 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e528956ee4704ac0b5ce4c62826b71ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:07.460002 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:07.460279 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:07.460548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:07.462864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:07.475411 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:07.477104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:07.487849 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1427/2848] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:17.483922 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:25.046908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:25.048314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:25.107105 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f42506-4f98-48c2-8d8c-73644d1811a3 service swift] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:25.110615 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:25.111205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:25.111508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:25.111774 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:25.113505 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:25.169908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:25.171320 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:25.180039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1422/2849] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:25 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:30.656911 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:30.658538 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:30.720940 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f4e56c-d422-4cc8-a256-34f09afdc96a service nova] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:30.725409 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:30.726027 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:30.726308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:30.726578 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] PATH_INFO: `/v3/users/9365ac5a4b274e72b01afca8906ac79a` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:30.727683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9365ac5a4b274e72b01afca8906ac79a)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:30.730233 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:30.775000 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9365ac5a4b274e72b01afca8906ac79a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:30.787617 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9365ac5a4b274e72b01afca8906ac79a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:30.795797 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9365ac5a4b274e72b01afca8906ac79a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:30.796329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9365ac5a4b274e72b01afca8906ac79a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:30.797020 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9365ac5a4b274e72b01afca8906ac79a'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:30.812197 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1428/2850] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:30 2022] DELETE /identity/v3/users/9365ac5a4b274e72b01afca8906ac79a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:30.997704 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:31.009886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338044db-c01c-41d6-a1f3-a26a6a3594c4 service cinder] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:31.071606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:31.072512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:31.072892 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:31.073181 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] PATH_INFO: `/v3/projects/50e5b07fbfdd4ed190aee4702194a5fa` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:31.074289 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50e5b07fbfdd4ed190aee4702194a5fa)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:31.076162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:31.126696 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50e5b07fbfdd4ed190aee4702194a5fa', 'request_id': 'req-e1edfba2-14e1-46ee-a038-eaba5d974c52'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:31.133799 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50e5b07fbfdd4ed190aee4702194a5fa', 'request_id': 'req-e1edfba2-14e1-46ee-a038-eaba5d974c52'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:31.134937 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:31.140562 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50e5b07fbfdd4ed190aee4702194a5fa', 'request_id': 'req-e1edfba2-14e1-46ee-a038-eaba5d974c52'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:31.152539 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] The token cache is being invalidate because project 50e5b07fbfdd4ed190aee4702194a5fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:31.152835 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:31.153524 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:31.155003 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1423/2851] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:30 2022] DELETE /identity/v3/projects/50e5b07fbfdd4ed190aee4702194a5fa => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:32.075521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:32.076969 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:32.131328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788c69b4-81f4-41ac-bd95-5d4f1bba7c32 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:32.162435 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:32.180643 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:32.181284 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:32.181559 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:32.181828 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:32.183541 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:32.243704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:32.245091 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:32.254487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1429/2852] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:35.421434 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:35.422908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:35.436454 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1edfba2-14e1-46ee-a038-eaba5d974c52 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:35.441348 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:35.442094 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:35.442310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:35.442578 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] PATH_INFO: `/v3/users/e077cbccb35940608bc2224c373c0b99` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:35.443703 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e077cbccb35940608bc2224c373c0b99)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:35.445633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:35.490898 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e077cbccb35940608bc2224c373c0b99'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.497508 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e077cbccb35940608bc2224c373c0b99'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.498155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e077cbccb35940608bc2224c373c0b99'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.498744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e077cbccb35940608bc2224c373c0b99'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.505515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ac370454-691b-42c2-a384-fb61535a67b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e077cbccb35940608bc2224c373c0b99'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.522224 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1424/2853] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:35 2022] DELETE /identity/v3/users/e077cbccb35940608bc2224c373c0b99 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:35.709080 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:35.719476 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076670cf-e248-47f3-b0af-88089edf5a67 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:35.776746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:35.777505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:35.777784 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:35.778057 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] PATH_INFO: `/v3/projects/0a769dbf20fa4526a8c63c4b631c94b1` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:35.779174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a769dbf20fa4526a8c63c4b631c94b1)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:35.781118 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:35.825929 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a769dbf20fa4526a8c63c4b631c94b1', 'request_id': 'req-a5462144-284f-4469-a1ac-13442e3a095e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.833012 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a769dbf20fa4526a8c63c4b631c94b1', 'request_id': 'req-a5462144-284f-4469-a1ac-13442e3a095e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.839052 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a769dbf20fa4526a8c63c4b631c94b1', 'request_id': 'req-a5462144-284f-4469-a1ac-13442e3a095e'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.849988 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] The token cache is being invalidate because project 0a769dbf20fa4526a8c63c4b631c94b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:35.850273 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.850908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-a5462144-284f-4469-a1ac-13442e3a095e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:35.852362 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1430/2854] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:35 2022] DELETE /identity/v3/projects/0a769dbf20fa4526a8c63c4b631c94b1 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:35.857309 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:35.857598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:35.857874 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:35.876963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:35.878160 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:35.935836 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1425/2855] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:35.951193 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:35.951480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:35.951751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:35.972251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:35.973482 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:35.982045 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1431/2856] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:35.986294 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:35.996327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58c3b9e-9d26-4d23-84b5-f94c8e7fe508 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.000885 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.001475 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.001754 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.002025 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.002658 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:22:36.002960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.004534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.015758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1426/2857] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:22:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:22:36.019882 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:36.028950 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79759b-18f9-42b0-a894-28c01d33bf14 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.033656 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.034245 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.034523 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.034792 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.035417 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.037330 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.042921 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:36.075252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1432/2858] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:36 2022] POST /identity/v3/projects => generated 365 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:36.079291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:36.089107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06fb90-8b41-43f4-97dd-1b0cd68903f4 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.135446 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:36.152361 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.153023 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.153304 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.153581 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.154213 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.156100 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.176329 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1427/2859] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:36 2022] POST /identity/v3/users => generated 283 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:36.180345 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:36.190621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed6dbb-0b84-4853-b27d-ff0b2d99da54 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.195155 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.195742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.196019 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.196290 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.196984 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.198564 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.208154 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1433/2860] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:36 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:36.211848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:36.222199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88711c30-c8ed-4ffc-969d-464dda5c5a2f admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.226805 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207342e7-358c-4add-a845-7673a12525b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.227394 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207342e7-358c-4add-a845-7673a12525b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.227676 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207342e7-358c-4add-a845-7673a12525b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.227950 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207342e7-358c-4add-a845-7673a12525b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.229198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207342e7-358c-4add-a845-7673a12525b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.230781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207342e7-358c-4add-a845-7673a12525b9 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.241677 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1428/2861] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:36.245813 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:36.256168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6375a5-ef36-408f-86b7-5017614ef2f7 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.265281 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.265928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.266203 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.266472 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] PATH_INFO: `/v3/projects/b58f8cae5e134df88551d19a9e82d3db/users/a10a84a33b0049f48ee0d6b22a57085d/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.281772 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b58f8cae5e134df88551d19a9e82d3db, user_id=a10a84a33b0049f48ee0d6b22a57085d, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.284335 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.297508 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1434/2862] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:36 2022] PUT /identity/v3/projects/b58f8cae5e134df88551d19a9e82d3db/users/a10a84a33b0049f48ee0d6b22a57085d/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:22:36.308579 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.308960 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.309240 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.339352 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] MFA Rules not processed for user `a10a84a33b0049f48ee0d6b22a57085d`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:36.340564 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:36.403565 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1429/2863] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:36 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:36.414480 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:36.416747 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:36.470765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b1dce-4f34-46d9-b502-3d552b73d545 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:36.474305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219ceb5-0fc3-4d8b-991d-9d86ae376b4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:36.474902 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219ceb5-0fc3-4d8b-991d-9d86ae376b4b service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:36.475163 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219ceb5-0fc3-4d8b-991d-9d86ae376b4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:36.475427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219ceb5-0fc3-4d8b-991d-9d86ae376b4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:36.484051 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219ceb5-0fc3-4d8b-991d-9d86ae376b4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:36.485505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219ceb5-0fc3-4d8b-991d-9d86ae376b4b service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:36.494177 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1435/2864] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:36 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:44.835244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:44.894743 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b9c35-a6d6-47cb-a1bc-b0d8b50731fb None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:44.899209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:44.899826 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:44.900709 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:44.901017 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] PATH_INFO: `/v3/users/a10a84a33b0049f48ee0d6b22a57085d` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:44.902133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a10a84a33b0049f48ee0d6b22a57085d)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:44.904035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:44.949030 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a10a84a33b0049f48ee0d6b22a57085d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:44.954370 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:44.956485 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a10a84a33b0049f48ee0d6b22a57085d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:44.957440 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a10a84a33b0049f48ee0d6b22a57085d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:44.958398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a10a84a33b0049f48ee0d6b22a57085d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:44.966535 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a10a84a33b0049f48ee0d6b22a57085d'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:44.983666 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1430/2865] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:44 2022] DELETE /identity/v3/users/a10a84a33b0049f48ee0d6b22a57085d => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:44.988021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:44.988522 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:44.988634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.006780 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:45.007953 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:45.064736 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1436/2866] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:45.074811 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.076186 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:45.143411 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e1085-b01e-42ae-a25f-85070e10e608 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.147015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2362448c-2195-45ef-829e-c3c1ac3cde0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.147735 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2362448c-2195-45ef-829e-c3c1ac3cde0a service neutron] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.148008 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2362448c-2195-45ef-829e-c3c1ac3cde0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.148277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2362448c-2195-45ef-829e-c3c1ac3cde0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.157291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2362448c-2195-45ef-829e-c3c1ac3cde0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.158708 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2362448c-2195-45ef-829e-c3c1ac3cde0a service neutron] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.170512 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1431/2867] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:45.440614 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.442075 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:45.455689 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c5d5a3-4a15-4b67-a3be-68e0a7a020da None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.460521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.461208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.461483 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.461755 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] PATH_INFO: `/v3/projects/b58f8cae5e134df88551d19a9e82d3db` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.462905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b58f8cae5e134df88551d19a9e82d3db)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.465970 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.506457 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:45.513329 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b58f8cae5e134df88551d19a9e82d3db', 'request_id': 'req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:45.520257 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b58f8cae5e134df88551d19a9e82d3db', 'request_id': 'req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:45.525625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b58f8cae5e134df88551d19a9e82d3db', 'request_id': 'req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:45.536342 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] The token cache is being invalidate because project b58f8cae5e134df88551d19a9e82d3db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81544) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:45.536631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:45.537305 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-cc704a03-cc7b-4e1a-8d99-6c063ae774b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:45.538782 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1437/2868] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:45 2022] DELETE /identity/v3/projects/b58f8cae5e134df88551d19a9e82d3db => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:45.544158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.544448 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.544722 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.565397 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:45.566597 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:45.624527 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1432/2869] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:45.632337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.632621 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.632917 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.651941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:45.653087 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:45.660742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1438/2870] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:45.664780 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.674766 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04009cfe-06b0-43cf-99b7-cb2cc072cdd3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.679247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.679839 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.680117 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.680389 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] PATH_INFO: `/v3/domains` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.681652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81543) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:22:45.681976 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.683554 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.694865 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1433/2871] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:22:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:22:45.698938 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.707816 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dd57a5-3c59-4700-b72f-178337b52a68 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.712376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.712989 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.713265 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.713530 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.714142 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.716005 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.742144 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1439/2872] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:45 2022] POST /identity/v3/projects => generated 403 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:45.746979 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.756910 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61412058-c9dd-40fa-ab30-9d95cd396e1e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.812067 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.812656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.812963 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.813236 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.813848 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.815983 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.824598 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:45.837664 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1434/2873] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:45 2022] POST /identity/v3/users => generated 302 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:45.841794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.851837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a72416-8e03-408f-b4bc-be5f43b9ebda admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.857253 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.857847 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.858140 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.858390 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.859017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.860593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.871304 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1440/2874] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:45.875133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.884978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a5893-5aea-42a6-872e-1cd5e3655da9 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.889553 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fc0522-752b-4c99-b261-287657d1e6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.890143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fc0522-752b-4c99-b261-287657d1e6df admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.890423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fc0522-752b-4c99-b261-287657d1e6df admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.890695 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fc0522-752b-4c99-b261-287657d1e6df admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.891325 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fc0522-752b-4c99-b261-287657d1e6df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.892917 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fc0522-752b-4c99-b261-287657d1e6df admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.904256 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1435/2875] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:45.907527 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:45.917951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbdf815-fbac-4e9e-9e7f-f1c8005ce6b4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:45.922494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:45.923125 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.923405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.923679 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] PATH_INFO: `/v3/projects/1ca851b7e9c243018f244582766aa067/users/3bb7ddda53c8457497c033e5b5d12dbd/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.937851 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ca851b7e9c243018f244582766aa067, user_id=3bb7ddda53c8457497c033e5b5d12dbd, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:45.940771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:45.954065 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1441/2876] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:45 2022] PUT /identity/v3/projects/1ca851b7e9c243018f244582766aa067/users/3bb7ddda53c8457497c033e5b5d12dbd/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:45.964859 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:45.965245 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:45.965589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:45.992846 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] MFA Rules not processed for user `3bb7ddda53c8457497c033e5b5d12dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:45.994173 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:46.055254 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1436/2877] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:45 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:22:46.059686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.121017 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d54521-8fd0-4da1-a0bc-bf7794f3ef90 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.125625 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.126218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.126492 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.126760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.127376 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.129308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.155501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1442/2878] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:46 2022] POST /identity/v3/projects => generated 405 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:46.159555 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.170638 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789ac7-3c74-467c-b954-63676666a5a3 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.224932 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.225534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.225818 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.226093 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] PATH_INFO: `/v3/users` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.226746 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.229318 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.249529 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1437/2879] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:46 2022] POST /identity/v3/users => generated 309 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:46.253628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.264512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd8fa5-e0f0-4d04-9e37-583334735dc0 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.269260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.269852 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.270128 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.270396 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.271056 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.272668 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.283812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1443/2880] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:46.287778 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.297596 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcbd203-c190-447b-8b65-ebca0c6d4d10 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.302167 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.302812 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.303095 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.303371 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] PATH_INFO: `/v3/projects/724f30d66b3b492ea76566ac7d234182/users/65419884b0d246eab13f249eb4f1382d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.319493 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] RBAC: Authorizing `identity:create_grant(project_id=724f30d66b3b492ea76566ac7d234182, user_id=65419884b0d246eab13f249eb4f1382d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.322691 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.334864 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1438/2881] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:46 2022] PUT /identity/v3/projects/724f30d66b3b492ea76566ac7d234182/users/65419884b0d246eab13f249eb4f1382d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:46.339348 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.349424 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a0d2f-1036-4746-8b09-1872522532dd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.387414 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:46.408048 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce1ae53-f0c2-4a2e-86a2-4430b75015b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.408589 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce1ae53-f0c2-4a2e-86a2-4430b75015b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.408893 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce1ae53-f0c2-4a2e-86a2-4430b75015b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.409173 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce1ae53-f0c2-4a2e-86a2-4430b75015b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.409804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce1ae53-f0c2-4a2e-86a2-4430b75015b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.411388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce1ae53-f0c2-4a2e-86a2-4430b75015b5 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.420837 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1444/2882] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:46 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:46.424962 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.435036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9dec1b-45f0-4c3a-9d0d-60c288530714 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.439841 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.440472 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.440747 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.441053 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] PATH_INFO: `/v3/projects/724f30d66b3b492ea76566ac7d234182/users/65419884b0d246eab13f249eb4f1382d/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.442978 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=724f30d66b3b492ea76566ac7d234182, user_id=65419884b0d246eab13f249eb4f1382d, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.445614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.456219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1439/2883] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:46 2022] PUT /identity/v3/projects/724f30d66b3b492ea76566ac7d234182/users/65419884b0d246eab13f249eb4f1382d/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:46.468354 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.468642 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.468951 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.495941 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] MFA Rules not processed for user `65419884b0d246eab13f249eb4f1382d`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:46.497182 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:46.565223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1445/2884] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:46 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:46.576222 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.577942 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:46.604230 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:46.635589 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d77d67b-dfc8-4821-808a-764b05bda3d5 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.639198 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7884415f-9f7a-4e78-9174-1675e125b19b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.639797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7884415f-9f7a-4e78-9174-1675e125b19b service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.640073 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7884415f-9f7a-4e78-9174-1675e125b19b service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.640337 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7884415f-9f7a-4e78-9174-1675e125b19b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.649080 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7884415f-9f7a-4e78-9174-1675e125b19b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.650492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7884415f-9f7a-4e78-9174-1675e125b19b service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.659619 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1440/2885] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:46 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:46.792610 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:46.853519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46fb528-0c40-47de-8f92-829ef8691ccb None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:46.857990 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:46.858602 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.858878 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.859146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] PATH_INFO: `/v3/users/3bb7ddda53c8457497c033e5b5d12dbd` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.860246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bb7ddda53c8457497c033e5b5d12dbd)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:46.862165 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:46.907487 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bb7ddda53c8457497c033e5b5d12dbd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:46.918043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:46.919187 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bb7ddda53c8457497c033e5b5d12dbd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:46.927036 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bb7ddda53c8457497c033e5b5d12dbd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:46.927693 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bb7ddda53c8457497c033e5b5d12dbd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:46.928295 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bb7ddda53c8457497c033e5b5d12dbd'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:46.941061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1446/2886] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:46 2022] DELETE /identity/v3/users/3bb7ddda53c8457497c033e5b5d12dbd => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:46.945531 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:46.946021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:46.946147 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:46.966775 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:46.968248 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:47.024507 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1441/2887] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:47.034216 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:47.035593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:47.090918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b38e14-5da7-49b3-884d-efba42905c1c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:47.094543 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:47.095129 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:47.095405 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:47.095674 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:47.104887 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:47.106308 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:47.115687 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1447/2888] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:47.400526 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:47.401969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:47.413955 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d3f8f-6c4e-4fd8-a26a-445411b7d2ee None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:47.419469 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:47.420112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:47.420395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:47.420665 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] PATH_INFO: `/v3/projects/1ca851b7e9c243018f244582766aa067` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:47.422042 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ca851b7e9c243018f244582766aa067)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:47.423959 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:47.472021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ca851b7e9c243018f244582766aa067', 'request_id': 'req-cede8126-af40-435c-86ef-b05bffeb6bdc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.478866 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ca851b7e9c243018f244582766aa067', 'request_id': 'req-cede8126-af40-435c-86ef-b05bffeb6bdc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.485133 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ca851b7e9c243018f244582766aa067', 'request_id': 'req-cede8126-af40-435c-86ef-b05bffeb6bdc'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.495603 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] The token cache is being invalidate because project 1ca851b7e9c243018f244582766aa067 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:47.495886 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.496499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.497996 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1442/2889] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:47 2022] DELETE /identity/v3/projects/1ca851b7e9c243018f244582766aa067 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:47.501928 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:47.511976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1828bf2-1e1f-4bf4-86e9-7b67c7a6d351 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:47.566901 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:47.567539 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:47.567812 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:47.568078 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] PATH_INFO: `/v3/users/65419884b0d246eab13f249eb4f1382d` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:47.569260 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65419884b0d246eab13f249eb4f1382d)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:47.571252 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:47.615905 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65419884b0d246eab13f249eb4f1382d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.627858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65419884b0d246eab13f249eb4f1382d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.634993 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65419884b0d246eab13f249eb4f1382d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.635648 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65419884b0d246eab13f249eb4f1382d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.636251 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-16cebedb-da0a-4e8a-bf5c-8781cce49aa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65419884b0d246eab13f249eb4f1382d'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:47.648746 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1448/2890] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:47 2022] DELETE /identity/v3/users/65419884b0d246eab13f249eb4f1382d => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:47.916244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:47.917715 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:47.930567 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cede8126-af40-435c-86ef-b05bffeb6bdc admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:47.974588 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:47.987614 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:47.988237 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:47.988515 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:47.988785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] PATH_INFO: `/v3/projects/724f30d66b3b492ea76566ac7d234182` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:47.989900 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=724f30d66b3b492ea76566ac7d234182)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:47.991782 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.035006 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '724f30d66b3b492ea76566ac7d234182', 'request_id': 'req-ba53153f-8ce1-4ae4-b702-104b8985e7e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:48.043146 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '724f30d66b3b492ea76566ac7d234182', 'request_id': 'req-ba53153f-8ce1-4ae4-b702-104b8985e7e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:48.049122 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '724f30d66b3b492ea76566ac7d234182', 'request_id': 'req-ba53153f-8ce1-4ae4-b702-104b8985e7e5'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:48.060078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] The token cache is being invalidate because project 724f30d66b3b492ea76566ac7d234182 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:48.060367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:48.061035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ba53153f-8ce1-4ae4-b702-104b8985e7e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:48.062492 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1443/2891] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:47 2022] DELETE /identity/v3/projects/724f30d66b3b492ea76566ac7d234182 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:48.067616 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.067899 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.068168 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.075962 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:48.089494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:48.090683 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.147116 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1449/2892] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:48.155758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.156052 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.156321 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.176452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:48.177688 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.185421 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1444/2893] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:48.189874 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.199455 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee872427-af3e-45a1-ac74-de41006e8a56 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.203925 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.204513 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.204788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.205084 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.205744 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:22:48.206042 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.207613 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.219680 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1450/2894] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:22:48.223845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.233489 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42dbbd8-dd37-4fd1-8171-487d571f1cc2 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.238155 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.238744 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.239021 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.239290 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.239914 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.241828 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.268235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1445/2895] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:22:48.272223 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.282503 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4277910-fecb-45d1-a706-7d84188fd9d4 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.337684 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.338219 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.338495 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.338760 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.339366 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.341285 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.362423 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1451/2896] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:48.366534 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.376649 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06a5dff-23d2-4cc6-b6c2-6498807c695b admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.381202 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.381781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.382060 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.382328 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.382958 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.384516 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.394078 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1446/2897] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:48.397721 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.408403 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502afa9-7a21-4d8c-a7b5-6211ccc70b6c admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.413246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56acd343-6032-4ae8-8ea2-a3631999bd95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.413864 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56acd343-6032-4ae8-8ea2-a3631999bd95 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.414146 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56acd343-6032-4ae8-8ea2-a3631999bd95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.414419 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56acd343-6032-4ae8-8ea2-a3631999bd95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.415061 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56acd343-6032-4ae8-8ea2-a3631999bd95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.417369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56acd343-6032-4ae8-8ea2-a3631999bd95 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.427598 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1452/2898] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:48.431667 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.441569 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64aec473-ddd1-44c3-b637-d1bc485a9f49 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.446056 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.446686 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.446965 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.447235 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] PATH_INFO: `/v3/projects/48f95a14f5294f6ca8d13e44d0b05f9d/users/b77bd5c7dd93493e861dc4372b480176/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.461461 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] RBAC: Authorizing `identity:create_grant(project_id=48f95a14f5294f6ca8d13e44d0b05f9d, user_id=b77bd5c7dd93493e861dc4372b480176, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.464557 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.477407 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1447/2899] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:48 2022] PUT /identity/v3/projects/48f95a14f5294f6ca8d13e44d0b05f9d/users/b77bd5c7dd93493e861dc4372b480176/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:48.489214 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.489498 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.489771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.518433 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] MFA Rules not processed for user `b77bd5c7dd93493e861dc4372b480176`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:48.519628 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.578021 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:48.588952 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1453/2900] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:48.594277 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.653310 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd0c485-429f-41cb-a080-665e6119b08c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.657911 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.658496 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.658772 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.659039 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.659668 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.661582 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.681476 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:48.688377 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1448/2901] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/projects => generated 383 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:48.692450 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.703268 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ce1537-c73b-4061-8012-08262f8ee452 None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.756795 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.757505 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.757845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.758119 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.758779 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.761212 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.781715 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1454/2902] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/users => generated 298 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:48.785685 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.795783 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8d2393-8e5c-48b2-9bf2-97c6b24e2ec0 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.800367 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.800994 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.801275 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.801548 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.802182 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.803758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.816036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1449/2903] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:48.820238 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.830787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e30de9d-610b-44f6-8774-cafc14082374 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.835440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.836098 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.836379 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.836653 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] PATH_INFO: `/v3/projects/8195df2089c4448f9cc646c7af191f40/users/6cd8bc2b30744f97b6f9b3e1a033f8ac/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.852655 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8195df2089c4448f9cc646c7af191f40, user_id=6cd8bc2b30744f97b6f9b3e1a033f8ac, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.855440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.868072 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1455/2904] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:48 2022] PUT /identity/v3/projects/8195df2089c4448f9cc646c7af191f40/users/6cd8bc2b30744f97b6f9b3e1a033f8ac/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:48.872518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.882262 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ea4e9c-3c89-4865-b0a5-33d18be94b4e admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.937758 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e502118e-e1d3-426e-9ab3-ec91cc3a9ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.938354 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e502118e-e1d3-426e-9ab3-ec91cc3a9ab6 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.938634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e502118e-e1d3-426e-9ab3-ec91cc3a9ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.939158 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e502118e-e1d3-426e-9ab3-ec91cc3a9ab6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.939785 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e502118e-e1d3-426e-9ab3-ec91cc3a9ab6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.941420 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e502118e-e1d3-426e-9ab3-ec91cc3a9ab6 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.951295 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1450/2905] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:48.955443 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.965622 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91edb7-82be-40a8-8fc0-66b748f4c2b2 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.970167 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.970885 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.971226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.971526 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] PATH_INFO: `/v3/projects/8195df2089c4448f9cc646c7af191f40/users/6cd8bc2b30744f97b6f9b3e1a033f8ac/roles/619e79e7bb94488780bd4da040d61c34` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.973436 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=8195df2089c4448f9cc646c7af191f40, user_id=6cd8bc2b30744f97b6f9b3e1a033f8ac, role_id=619e79e7bb94488780bd4da040d61c34)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.976037 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.986106 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1456/2906] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:48 2022] PUT /identity/v3/projects/8195df2089c4448f9cc646c7af191f40/users/6cd8bc2b30744f97b6f9b3e1a033f8ac/roles/619e79e7bb94488780bd4da040d61c34 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:22:49.047399 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:49.048007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:49.048007 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:49.076602 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] MFA Rules not processed for user `6cd8bc2b30744f97b6f9b3e1a033f8ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:49.077840 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:49.151456 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1451/2907] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:49 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:49.161981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:49.163728 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:49.224569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b613b85-2cca-4bf5-9843-498a3d5819cd admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:49.228742 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:49.229364 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:49.229646 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:49.229918 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:49.283363 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:49.284791 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:49.293967 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1457/2908] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:53.489219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:53.490626 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:53.502369 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4713eed-6859-4a0f-9983-2190b88ba049 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:53.505930 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa27708-65e9-4772-b416-1c91244ff896 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:53.506518 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa27708-65e9-4772-b416-1c91244ff896 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:53.506793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa27708-65e9-4772-b416-1c91244ff896 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:53.507062 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa27708-65e9-4772-b416-1c91244ff896 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:53.515595 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa27708-65e9-4772-b416-1c91244ff896 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:53.517036 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa27708-65e9-4772-b416-1c91244ff896 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:53.526591 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1452/2909] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:53 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:58.403133 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:58.408341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:58.463807 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01944416-3678-4f7e-9be1-625422fa1e44 service cinder] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:58.468243 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:58.468853 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:58.469158 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:58.469427 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] PATH_INFO: `/v3/users/b77bd5c7dd93493e861dc4372b480176` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:58.470519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b77bd5c7dd93493e861dc4372b480176)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:58.472456 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:58.516300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b77bd5c7dd93493e861dc4372b480176'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:58.528593 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b77bd5c7dd93493e861dc4372b480176'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:58.535548 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b77bd5c7dd93493e861dc4372b480176'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:58.536200 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b77bd5c7dd93493e861dc4372b480176'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:58.536804 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b77bd5c7dd93493e861dc4372b480176'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:58.549030 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1458/2910] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:58 2022] DELETE /identity/v3/users/b77bd5c7dd93493e861dc4372b480176 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:58.553494 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:58.553986 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:58.554109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:58.573730 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:58.575011 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:58.623111 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=100 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:58.632522 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1453/2911] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:58.642713 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:58.644176 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:58.700031 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68626376-d6a5-46e3-bb9a-4167fed6cd64 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:58.703631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:58.704222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:58.704501 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:58.704771 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:58.713563 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:58.714981 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:58.724401 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1459/2912] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:58.984724 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:58.986205 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:59.000244 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57039cf-4b53-4cf8-8037-109da4648946 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.006952 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.007656 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.007969 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.008327 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] PATH_INFO: `/v3/projects/48f95a14f5294f6ca8d13e44d0b05f9d` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.009486 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] RBAC: Authorizing `identity:delete_project(project_id=48f95a14f5294f6ca8d13e44d0b05f9d)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.011499 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.059286 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48f95a14f5294f6ca8d13e44d0b05f9d', 'request_id': 'req-0bea0fc8-7eed-4f31-8ca1-95be0031304c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.066845 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48f95a14f5294f6ca8d13e44d0b05f9d', 'request_id': 'req-0bea0fc8-7eed-4f31-8ca1-95be0031304c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.073016 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48f95a14f5294f6ca8d13e44d0b05f9d', 'request_id': 'req-0bea0fc8-7eed-4f31-8ca1-95be0031304c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.083015 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] The token cache is being invalidate because project 48f95a14f5294f6ca8d13e44d0b05f9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:59.083291 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.083908 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.085398 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1454/2913] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:58 2022] DELETE /identity/v3/projects/48f95a14f5294f6ca8d13e44d0b05f9d => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:59.089577 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:59.099006 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b886aaf-d4ec-4055-932d-102a35f3a5e1 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.122706 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:59.157008 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.157626 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.157903 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.158174 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] PATH_INFO: `/v3/users/6cd8bc2b30744f97b6f9b3e1a033f8ac` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.159298 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cd8bc2b30744f97b6f9b3e1a033f8ac)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.161337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.206139 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cd8bc2b30744f97b6f9b3e1a033f8ac'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.217631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cd8bc2b30744f97b6f9b3e1a033f8ac'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.224440 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cd8bc2b30744f97b6f9b3e1a033f8ac'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.225162 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cd8bc2b30744f97b6f9b3e1a033f8ac'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.225794 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-c40bac02-ee52-4d31-a892-e5215ced5665 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cd8bc2b30744f97b6f9b3e1a033f8ac'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.238860 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1460/2914] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:59 2022] DELETE /identity/v3/users/6cd8bc2b30744f97b6f9b3e1a033f8ac => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:59.509026 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:59.510509 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:59.525163 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bea0fc8-7eed-4f31-8ca1-95be0031304c admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.582314 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.582971 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.583273 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.583506 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] PATH_INFO: `/v3/projects/8195df2089c4448f9cc646c7af191f40` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.584653 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8195df2089c4448f9cc646c7af191f40)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.586635 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.631604 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8195df2089c4448f9cc646c7af191f40', 'request_id': 'req-ea9d15da-d34d-4cc3-9221-69858cb6d42c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.638438 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8195df2089c4448f9cc646c7af191f40', 'request_id': 'req-ea9d15da-d34d-4cc3-9221-69858cb6d42c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.644742 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8195df2089c4448f9cc646c7af191f40', 'request_id': 'req-ea9d15da-d34d-4cc3-9221-69858cb6d42c'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.655340 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] The token cache is being invalidate because project 8195df2089c4448f9cc646c7af191f40 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:59.655620 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.656238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-ea9d15da-d34d-4cc3-9221-69858cb6d42c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:59.657726 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1455/2915] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:59 2022] DELETE /identity/v3/projects/8195df2089c4448f9cc646c7af191f40 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:59.663765 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.664247 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.664350 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.685516 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:59.686634 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:59.743521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1461/2916] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:59.751519 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.751808 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.752082 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.771606 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:59.772797 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:59.781227 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1456/2917] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:22:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:59.785341 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:59.795043 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c8f63b-42b6-48b1-ae44-f8ac83903f2f None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.799631 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.800519 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.800800 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.801109 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] PATH_INFO: `/v3/domains` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.801785 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81544) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:22:59.802086 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.803671 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.815300 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1462/2918] 10.222.0.27 () {58 vars in 1153 bytes} [Thu Sep 29 02:22:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:22:59.819647 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:59.825734 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:59.829425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac13e338-f087-4215-91e1-4b16e2cee658 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.835162 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.835858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.836308 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.836793 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.838059 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.840109 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.866239 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1457/2919] 10.222.0.27 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:59 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:22:59.870512 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:59.880208 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2bf2ee-148b-4b00-93f4-311c73ee2709 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.909226 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:59.935701 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.936246 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.936528 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.936806 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] PATH_INFO: `/v3/users` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.937463 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.939686 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.960477 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1463/2920] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:59 2022] POST /identity/v3/users => generated 292 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:59.965112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:59.975378 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff6fc5-e9b7-421c-ae9f-5391e0cdef92 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:59.980048 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:59.980636 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:59.980946 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:59.981219 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] PATH_INFO: `/v3/roles` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:59.981858 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:59.983426 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:59.992989 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1458/2921] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:59 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:59.996788 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:00.006494 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17966f3-41b3-4049-b76d-dc2d77362074 admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:00.011556 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f6d3d3-fd3e-485c-8bcf-c0f7f7f1047d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:00.012114 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6d3d3-fd3e-485c-8bcf-c0f7f7f1047d admin admin] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:00.012393 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6d3d3-fd3e-485c-8bcf-c0f7f7f1047d admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:00.012663 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f6d3d3-fd3e-485c-8bcf-c0f7f7f1047d admin admin] PATH_INFO: `/v3/roles` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:00.013328 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f6d3d3-fd3e-485c-8bcf-c0f7f7f1047d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:00.014896 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f6d3d3-fd3e-485c-8bcf-c0f7f7f1047d admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:00.025704 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1464/2922] 10.222.0.27 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:23:00.029642 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:00.039285 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338d06f-304c-476d-830e-62643882b201 admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:00.047779 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:00.048678 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:00.049138 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:00.049514 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] PATH_INFO: `/v3/projects/7cddf7251ce14ffb902122561fdd9306/users/5a5300840d60474881e96e05626ff238/roles/ce237b90aca34abead4f05dfc46a1ec4` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:00.063856 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cddf7251ce14ffb902122561fdd9306, user_id=5a5300840d60474881e96e05626ff238, role_id=ce237b90aca34abead4f05dfc46a1ec4)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:00.066781 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:00.079739 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1459/2923] 10.222.0.27 () {60 vars in 1483 bytes} [Thu Sep 29 02:23:00 2022] PUT /identity/v3/projects/7cddf7251ce14ffb902122561fdd9306/users/5a5300840d60474881e96e05626ff238/roles/ce237b90aca34abead4f05dfc46a1ec4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:23:00.092751 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] REQUEST_METHOD: `POST` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:00.093067 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:00.093373 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:00.120858 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.auth.core [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] MFA Rules not processed for user `5a5300840d60474881e96e05626ff238`. Rule list: `[]` (Enabled: `True`). {{(pid=81544) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:00.122071 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:00.184388 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1465/2924] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:23:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:00.195790 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:00.198085 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:00.253429 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88630216-cc66-4fa2-856b-137280c1d1bb admin admin] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:00.257035 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e16f00-d847-4a6e-aa4a-32141eaf20d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09c034ad1ccd45e78a4b3aeab2129227', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:00.257631 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e16f00-d847-4a6e-aa4a-32141eaf20d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:00.257909 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e16f00-d847-4a6e-aa4a-32141eaf20d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:00.258179 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e16f00-d847-4a6e-aa4a-32141eaf20d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:00.266697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e16f00-d847-4a6e-aa4a-32141eaf20d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:00.268107 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e16f00-d847-4a6e-aa4a-32141eaf20d1 service cinder] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:00.276609 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1460/2925] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:07.822757 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:07.882868 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba5d1-4cfb-44c4-ac48-5bc84b58840e None None] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:07.887218 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:07.887830 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:07.888104 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:07.888369 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] PATH_INFO: `/v3/users/5a5300840d60474881e96e05626ff238` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:07.889468 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a5300840d60474881e96e05626ff238)` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:07.891374 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:07.933787 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:07.936735 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a5300840d60474881e96e05626ff238'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:07.950096 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a5300840d60474881e96e05626ff238'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:07.956908 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a5300840d60474881e96e05626ff238'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:07.957579 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a5300840d60474881e96e05626ff238'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:07.958185 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.notifications [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a5300840d60474881e96e05626ff238'} {{(pid=81544) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:07.970337 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1466/2926] 10.222.0.27 () {58 vars in 1224 bytes} [Thu Sep 29 02:23:07 2022] DELETE /identity/v3/users/5a5300840d60474881e96e05626ff238 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:23:07.974967 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] REQUEST_METHOD: `POST` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:07.975452 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:07.975545 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:07.995425 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.auth.core [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] MFA Rules not processed for user `90cfae9be7734268a0ec1a4bc2e21b77`. Rule list: `[]` (Enabled: `True`). {{(pid=81543) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:07.996652 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:08.052209 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1461/2927] 10.222.0.27 () {58 vars in 958 bytes} [Thu Sep 29 02:23:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:08.062521 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:08.064188 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:08.120380 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6decc5-1781-4c34-88e6-2ab4b98bb8fc admin admin] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:08.123976 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afdc21c25f734a51aa520e7d4e53455b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:08.124569 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:08.124845 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:08.125143 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:08.133945 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:08.135309 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:08.144808 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1467/2928] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:08.405414 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] Authenticating user token {{(pid=81543) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:08.406889 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.fernet_utils [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81543) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:08.419770 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3095105b-2dbc-456c-83c3-72244aeef7a9 None None] Validating token access rules against request {{(pid=81543) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:08.424395 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90cfae9be7734268a0ec1a4bc2e21b77', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '304c048daacb48678f2d64fc933b378e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81543) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:08.425143 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:08.425423 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:08.425697 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] PATH_INFO: `/v3/projects/7cddf7251ce14ffb902122561fdd9306` {{(pid=81543) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:08.426817 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cddf7251ce14ffb902122561fdd9306)` {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:08.428915 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] RBAC: Authorization granted {{(pid=81543) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:08.474630 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:08.478633 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cddf7251ce14ffb902122561fdd9306', 'request_id': 'req-9d2408b5-261b-4f24-99e3-b99fee7f2735'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:08.486634 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cddf7251ce14ffb902122561fdd9306', 'request_id': 'req-9d2408b5-261b-4f24-99e3-b99fee7f2735'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:08.493112 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cddf7251ce14ffb902122561fdd9306', 'request_id': 'req-9d2408b5-261b-4f24-99e3-b99fee7f2735'} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:08.503247 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] The token cache is being invalidate because project 7cddf7251ce14ffb902122561fdd9306 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81543) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:23:08.503533 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:08.504199 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG keystone.notifications [None req-9d2408b5-261b-4f24-99e3-b99fee7f2735 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81543) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:08.505748 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: [pid: 81543|app: 0|req: 1462/2929] 10.222.0.27 () {58 vars in 1233 bytes} [Thu Sep 29 02:23:08 2022] DELETE /identity/v3/projects/7cddf7251ce14ffb902122561fdd9306 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:23:18.140678 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:DELETE=4,keystone:SELECT=39,keystone:INSERT=1 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:18.500238 dv-cinder-740146-21-iscsi devstack@keystone.service[81543]: DEBUG dbcounter [-] [81543] Writing DB stats keystone:DELETE=4,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81543) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:36.528691 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] Authenticating user token {{(pid=81544) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.530132 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:36.601016 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce574a-53e9-472c-9966-ab60f1200bf0 service neutron] Validating token access rules against request {{(pid=81544) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.658076 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f929f94af045445bb0a608dee02a7c5e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2fb848d359fb4361b7724b736e0f5333', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81544) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.658673 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] REQUEST_METHOD: `GET` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.658948 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] SCRIPT_NAME: `/identity` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.659222 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81544) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.660943 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.fernet_utils [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81544) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:36.671979 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.673574 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c0771-c4b0-4059-be26-46a5cc70eb7d service placement] RBAC: Authorization granted {{(pid=81544) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.683254 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: [pid: 81544|app: 0|req: 1468/2930] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:46.679009 dv-cinder-740146-21-iscsi devstack@keystone.service[81544]: DEBUG dbcounter [-] [81544] Writing DB stats keystone:SELECT=58 {{(pid=81544) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}